[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:07 dist-test-slave-3ffd krb5kdc[16583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:07 dist-test-slave-3ffd krb5kdc[16583](info): set up 2 sockets
Jun 19 16:55:07 dist-test-slave-3ffd krb5kdc[16583](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 16:55:08.216765 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:08 dist-test-slave-3ffd krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:08.408430 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36993
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:08.783280 16596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:08.783607 16596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:08.783697 16596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:08.791646 16596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:08.791785 16596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:08.791832 16596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:08.791893 16596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:08.791937 16596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:08.802877 16596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:36993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:08.804258 16596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:08.805644 16596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:08.819674 16602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:08.819746 16604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:08.820045 16596 server_base.cc:1034] running on GCE node
W20260619 16:55:08.819713 16601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:08.821460 16596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:08.823278 16596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:08.824559 16596 hybrid_clock.cc:648] HybridClock initialized: now 1781888108824446 us; error 101 us; skew 500 ppm
I20260619 16:55:08.824977 16596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:08 dist-test-slave-3ffd krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888108, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:08.834879 16596 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:08.836838 16596 webserver.cc:460] Webserver started at http://127.16.48.62:35149/ using document root <none> and password file <none>
I20260619 16:55:08.837733 16596 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:08.837904 16596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:08.838276 16596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:08.841902 16596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "8547ce4d05934f5f856be5b834e991d1"
format_stamp: "Formatted at 2026-06-19 16:55:08 on dist-test-slave-3ffd"
server_key: "71b794616c01164a8aa4b6654f47dae2"
server_key_iv: "2b003d0e6b3fa54594f5804855e93d05"
server_key_version: "encryptionkey@0"
I20260619 16:55:08.842795 16596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "8547ce4d05934f5f856be5b834e991d1"
format_stamp: "Formatted at 2026-06-19 16:55:08 on dist-test-slave-3ffd"
server_key: "71b794616c01164a8aa4b6654f47dae2"
server_key_iv: "2b003d0e6b3fa54594f5804855e93d05"
server_key_version: "encryptionkey@0"
I20260619 16:55:08.849174 16596 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 16:55:08.853296 16611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:08.854977 16596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:55:08.855209 16596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "8547ce4d05934f5f856be5b834e991d1"
format_stamp: "Formatted at 2026-06-19 16:55:08 on dist-test-slave-3ffd"
server_key: "71b794616c01164a8aa4b6654f47dae2"
server_key_iv: "2b003d0e6b3fa54594f5804855e93d05"
server_key_version: "encryptionkey@0"
I20260619 16:55:08.855450 16596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:08.874109 16596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:08.878890 16596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:08.879240 16596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:08.900521 16596 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:36993
I20260619 16:55:08.901018 16662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:36993 every 8 connection(s)
I20260619 16:55:08.902407 16596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:08.906805 16663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:08.912621 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 16596
I20260619 16:55:08.912986 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:08.915591 16576 external_mini_cluster.cc:1428] Setting key 5b9dbe4b462b3c60a08e9c4f656df0c8
I20260619 16:55:08.920809 16663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1: Bootstrap starting.
I20260619 16:55:08.926275 16663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:08.927860 16663 log.cc:824] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:08.931767 16663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1: No bootstrap required, opened a new log
I20260619 16:55:08.943840 16663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547ce4d05934f5f856be5b834e991d1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36993 } }
I20260619 16:55:08.944407 16663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:08.944545 16663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8547ce4d05934f5f856be5b834e991d1, State: Initialized, Role: FOLLOWER
I20260619 16:55:08.945312 16663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547ce4d05934f5f856be5b834e991d1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36993 } }
I20260619 16:55:08.945636 16663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:08.945784 16663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:08.945982 16663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:08.949146 16663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547ce4d05934f5f856be5b834e991d1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36993 } }
I20260619 16:55:08.949853 16663 leader_election.cc:302] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8547ce4d05934f5f856be5b834e991d1; no voters:
I20260619 16:55:08.950623 16663 leader_election.cc:288] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:08.950762 16668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:08.952572 16668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [term 1 LEADER]: Becoming Leader. State: Replica: 8547ce4d05934f5f856be5b834e991d1, State: Running, Role: LEADER
I20260619 16:55:08.953269 16668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547ce4d05934f5f856be5b834e991d1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36993 } }
I20260619 16:55:08.953810 16663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 16:55:08 dist-test-slave-3ffd krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:08.956406 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8547ce4d05934f5f856be5b834e991d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547ce4d05934f5f856be5b834e991d1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36993 } } }
I20260619 16:55:08.956815 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:08.957204 16670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8547ce4d05934f5f856be5b834e991d1. Latest consensus state: current_term: 1 leader_uuid: "8547ce4d05934f5f856be5b834e991d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547ce4d05934f5f856be5b834e991d1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36993 } } }
I20260619 16:55:08.957494 16670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:08.972150 16673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:08.974002 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:08.920931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.16.48.62:36993)
0619 16:55:08.922051 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:08.922081 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:08.924042 (+ 1961us) server_negotiation.cc:407] Connection header received
0619 16:55:08.938037 (+ 13995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:08.938081 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:08.938620 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:08.939048 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:08.943593 (+ 4545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:08.945281 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:08.948067 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:08.948893 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:08.949124 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:08.958604 (+ 9480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:08.958621 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:08.958651 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:08.958744 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:08.965349 (+ 6605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:08.967126 (+ 1777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:08.967143 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:08.967157 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:08.967497 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:08.968347 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:08.968354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:08.968360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:08.969413 (+ 1053us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:08.969799 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:08.972399 (+ 2600us) server_negotiation.cc:299] Negotiation successful
0619 16:55:08.972833 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":430,"threads_started":1}
I20260619 16:55:08.981163 16673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:08.994653 16673 catalog_manager.cc:1347] Generated new cluster ID: 5e8907fb95354f1591e5019ad3874361
I20260619 16:55:08.994814 16673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:09.008613 16673 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:09.010726 16673 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:09.021508 16673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8547ce4d05934f5f856be5b834e991d1: Generated new TSK 0
I20260619 16:55:09.022588 16673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:55:09.080783 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36993
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:09.460662 16690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:09.460966 16690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:09.461068 16690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:09.469223 16690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:09.469368 16690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:09.469486 16690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:09.479640 16690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:09.481004 16690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:09.482558 16690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:09.497866 16695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:09.497818 16698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:09.497784 16696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:09.498530 16690 server_base.cc:1034] running on GCE node
I20260619 16:55:09.499281 16690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:09.500409 16690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:09.501809 16690 hybrid_clock.cc:648] HybridClock initialized: now 1781888109501758 us; error 48 us; skew 500 ppm
I20260619 16:55:09.502189 16690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:09 dist-test-slave-3ffd krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:09.505704 16690 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:55:09.507277 16690 webserver.cc:460] Webserver started at http://127.16.48.1:40687/ using document root <none> and password file <none>
I20260619 16:55:09.508049 16690 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:09.508234 16690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:09.508579 16690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:09.511792 16690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "ae148934d2d64683b298bb035d42a9b8"
format_stamp: "Formatted at 2026-06-19 16:55:09 on dist-test-slave-3ffd"
server_key: "e8f53b4ea07a3c0f9f78f976e3e33d80"
server_key_iv: "ed4563bdfd3db33945c8e2123d33ce3c"
server_key_version: "encryptionkey@0"
I20260619 16:55:09.512743 16690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "ae148934d2d64683b298bb035d42a9b8"
format_stamp: "Formatted at 2026-06-19 16:55:09 on dist-test-slave-3ffd"
server_key: "e8f53b4ea07a3c0f9f78f976e3e33d80"
server_key_iv: "ed4563bdfd3db33945c8e2123d33ce3c"
server_key_version: "encryptionkey@0"
I20260619 16:55:09.518886 16690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:55:09.523113 16705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:09.524744 16690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:09.524959 16690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "ae148934d2d64683b298bb035d42a9b8"
format_stamp: "Formatted at 2026-06-19 16:55:09 on dist-test-slave-3ffd"
server_key: "e8f53b4ea07a3c0f9f78f976e3e33d80"
server_key_iv: "ed4563bdfd3db33945c8e2123d33ce3c"
server_key_version: "encryptionkey@0"
I20260619 16:55:09.525207 16690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:09.542209 16690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:09.547343 16690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:09.547737 16690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:09.548904 16690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:09.550434 16690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:09.550551 16690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:09.550729 16690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:09.550805 16690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:09.590181 16690 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:41413
I20260619 16:55:09.590236 16817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:41413 every 8 connection(s)
I20260619 16:55:09.591660 16690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:09.592854 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 16690
I20260619 16:55:09.593115 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:09.593734 16576 external_mini_cluster.cc:1428] Setting key c2df11648a501625b552d35cc9c917aa
Jun 19 16:55:09 dist-test-slave-3ffd krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:55:09.615619 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:09.594610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:51187 (local address 127.16.48.62:36993)
0619 16:55:09.595092 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:09.595099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:09.596498 (+ 1399us) server_negotiation.cc:407] Connection header received
0619 16:55:09.598377 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:09.598382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:09.598471 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:09.598612 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:09.600327 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:09.601073 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:09.602860 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:09.603666 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:09.603832 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:09.608345 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:09.608354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:09.608362 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:09.608416 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:09.611844 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:09.612975 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:09.612980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:09.612983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:09.613125 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:09.613633 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:09.613646 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:09.613652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:09.613969 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:09.614092 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:09.614992 (+ 900us) server_negotiation.cc:299] Negotiation successful
0619 16:55:09.615245 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260619 16:55:09.616783 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:09.595368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36993 (local address 127.16.48.1:51187)
0619 16:55:09.596334 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:09.596382 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:09.598066 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:09.598876 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:09.598886 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:09.599457 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:09.600074 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:09.600092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:09.601327 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:09.601332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:09.602661 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:09.602675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:09.603916 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:09.603921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:09.604218 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:09.604997 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:09.605021 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:09.607997 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:09.612019 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:09.612028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:09.612045 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:09.612798 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:09.613236 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:09.613240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:09.613242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:09.613451 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:09.614096 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:09.614103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:09.614554 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 16:55:09.614872 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 16:55:09.615456 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":354,"threads_started":1}
I20260619 16:55:09.618541 16818 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36993
I20260619 16:55:09.618908 16818 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:09.619755 16818 heartbeater.cc:511] Master 127.16.48.62:36993 requested a full tablet report, sending...
I20260619 16:55:09.622476 16628 ts_manager.cc:194] Registered new tserver with Master: ae148934d2d64683b298bb035d42a9b8 (127.16.48.1:41413)
I20260619 16:55:09.624868 16628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:51187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:55:09.639712 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36993
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:10.007567 16824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:10.007901 16824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:10.008002 16824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:10.016305 16824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:10.016426 16824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:10.016543 16824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:10.027004 16824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:10.028523 16824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:10.030210 16824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:10.046293 16829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:10.046459 16832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:10.046558 16830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:10.047314 16824 server_base.cc:1034] running on GCE node
I20260619 16:55:10.048015 16824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:10.049149 16824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:10.050400 16824 hybrid_clock.cc:648] HybridClock initialized: now 1781888110050343 us; error 55 us; skew 500 ppm
I20260619 16:55:10.050756 16824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:10 dist-test-slave-3ffd krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:10.054680 16824 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:55:10.056530 16824 webserver.cc:460] Webserver started at http://127.16.48.2:39865/ using document root <none> and password file <none>
I20260619 16:55:10.057320 16824 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:10.057476 16824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:10.057829 16824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:10.061381 16824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "4b45faf72ab048e5bdbaaee8336bb774"
format_stamp: "Formatted at 2026-06-19 16:55:10 on dist-test-slave-3ffd"
server_key: "3376edde9374be3fcd5cc70d6d012219"
server_key_iv: "725ab8f5ef5b0e24bef96e0a79a85c73"
server_key_version: "encryptionkey@0"
I20260619 16:55:10.062278 16824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "4b45faf72ab048e5bdbaaee8336bb774"
format_stamp: "Formatted at 2026-06-19 16:55:10 on dist-test-slave-3ffd"
server_key: "3376edde9374be3fcd5cc70d6d012219"
server_key_iv: "725ab8f5ef5b0e24bef96e0a79a85c73"
server_key_version: "encryptionkey@0"
I20260619 16:55:10.068218 16824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:55:10.072029 16839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:10.073657 16824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 16:55:10.073875 16824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "4b45faf72ab048e5bdbaaee8336bb774"
format_stamp: "Formatted at 2026-06-19 16:55:10 on dist-test-slave-3ffd"
server_key: "3376edde9374be3fcd5cc70d6d012219"
server_key_iv: "725ab8f5ef5b0e24bef96e0a79a85c73"
server_key_version: "encryptionkey@0"
I20260619 16:55:10.074100 16824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:10.097337 16824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:10.101693 16824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:10.102005 16824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:10.103075 16824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:10.104635 16824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:10.104750 16824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:10.104907 16824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:10.104981 16824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:10.145678 16824 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:37187
I20260619 16:55:10.145792 16951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:37187 every 8 connection(s)
I20260619 16:55:10.147248 16824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:10.149899 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 16824
I20260619 16:55:10.150148 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:10.150719 16576 external_mini_cluster.cc:1428] Setting key 195cc7f4b95e9415e776ed27472b0833
Jun 19 16:55:10 dist-test-slave-3ffd krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
I20260619 16:55:10.173763 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:10.150349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:51085 (local address 127.16.48.62:36993)
0619 16:55:10.150928 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:10.150937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:10.152601 (+ 1664us) server_negotiation.cc:407] Connection header received
0619 16:55:10.155568 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:10.155572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:10.155680 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:10.155781 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:10.158290 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.159539 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.161488 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.162319 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.162487 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:10.166886 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:10.166892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:10.166896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:10.166937 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:10.170213 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.171127 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.171132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.171135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.171283 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.171803 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.171810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.171818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.172178 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:10.172345 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:10.173114 (+ 769us) server_negotiation.cc:299] Negotiation successful
0619 16:55:10.173351 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":267,"threads_started":1}
I20260619 16:55:10.174844 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:10.150970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36993 (local address 127.16.48.2:51085)
0619 16:55:10.152405 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:10.152467 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:10.155162 (+ 2695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:10.156158 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:10.156175 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:10.157069 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:10.158007 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:10.158041 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.159750 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.159754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:10.161295 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:10.161310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.162559 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.162564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:10.162783 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:10.163572 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:10.163598 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:10.166505 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:10.170443 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:10.170453 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:10.170469 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:10.170968 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:10.171431 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:10.171437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:10.171441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:10.171666 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:10.172321 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:10.172330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:10.172732 (+ 402us) client_negotiation.cc:769] Sending connection context
0619 16:55:10.173023 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 16:55:10.173626 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":833,"threads_started":1}
I20260619 16:55:10.177040 16952 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36993
I20260619 16:55:10.177440 16952 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:10.178820 16952 heartbeater.cc:511] Master 127.16.48.62:36993 requested a full tablet report, sending...
I20260619 16:55:10.180598 16628 ts_manager.cc:194] Registered new tserver with Master: 4b45faf72ab048e5bdbaaee8336bb774 (127.16.48.2:37187)
I20260619 16:55:10.181443 16628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:51085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:55:10.195751 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36993
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:10.555521 16958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:10.555832 16958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:10.556001 16958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:10.564630 16958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:10.564766 16958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:10.564961 16958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:10.575456 16958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:10.576970 16958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:10.578464 16958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:10.594574 16966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:10.594549 16964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:10.595028 16963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:10.595808 16958 server_base.cc:1034] running on GCE node
I20260619 16:55:10.596537 16958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:10.597611 16958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:10.598860 16958 hybrid_clock.cc:648] HybridClock initialized: now 1781888110598816 us; error 50 us; skew 500 ppm
I20260619 16:55:10.599210 16958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:10 dist-test-slave-3ffd krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:10.603039 16958 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:55:10.604836 16958 webserver.cc:460] Webserver started at http://127.16.48.3:44327/ using document root <none> and password file <none>
I20260619 16:55:10.605618 16958 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:10.605785 16958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:10.606127 16958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:10.609359 16958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "ab30578100f444ceb2bd58d285434d08"
format_stamp: "Formatted at 2026-06-19 16:55:10 on dist-test-slave-3ffd"
server_key: "22351172898f7eb0b5a40c516c040411"
server_key_iv: "971e0ef61007d6afb9b633ba5d186c48"
server_key_version: "encryptionkey@0"
I20260619 16:55:10.610242 16958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "ab30578100f444ceb2bd58d285434d08"
format_stamp: "Formatted at 2026-06-19 16:55:10 on dist-test-slave-3ffd"
server_key: "22351172898f7eb0b5a40c516c040411"
server_key_iv: "971e0ef61007d6afb9b633ba5d186c48"
server_key_version: "encryptionkey@0"
I20260619 16:55:10.616214 16958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 16:55:10.620235 16973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:10.621913 16958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:10.622148 16958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "ab30578100f444ceb2bd58d285434d08"
format_stamp: "Formatted at 2026-06-19 16:55:10 on dist-test-slave-3ffd"
server_key: "22351172898f7eb0b5a40c516c040411"
server_key_iv: "971e0ef61007d6afb9b633ba5d186c48"
server_key_version: "encryptionkey@0"
I20260619 16:55:10.622396 16958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:10.629452 16818 heartbeater.cc:503] Master 127.16.48.62:36993 was elected leader, sending a full tablet report...
I20260619 16:55:10.669023 16958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:10.673635 16958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:10.673910 16958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:10.675036 16958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:10.676599 16958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:10.676677 16958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:10.676774 16958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:10.676817 16958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:10.718099 16958 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:39095
I20260619 16:55:10.718148 17085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:39095 every 8 connection(s)
I20260619 16:55:10.719455 16958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:10.726922 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 16958
I20260619 16:55:10.727160 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:10.727669 16576 external_mini_cluster.cc:1428] Setting key 081f3b58a3a5549a9f8e267b462e2e3b
Jun 19 16:55:10 dist-test-slave-3ffd krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:10.742301 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:10.722624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:37699 (local address 127.16.48.62:36993)
0619 16:55:10.723141 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:10.723146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:10.724158 (+ 1012us) server_negotiation.cc:407] Connection header received
0619 16:55:10.726211 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:10.726216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:10.726349 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:10.726495 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:10.728461 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.729688 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.731344 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.731745 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.731881 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:10.735729 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:10.735734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:10.735738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:10.735776 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:10.738363 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.739256 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.739261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.739263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.739395 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.739884 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.739888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.739890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.740705 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:10.740884 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:10.741664 (+ 780us) server_negotiation.cc:299] Negotiation successful
0619 16:55:10.741904 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":290,"threads_started":1}
I20260619 16:55:10.743628 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:10.723212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36993 (local address 127.16.48.3:37699)
0619 16:55:10.723941 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:10.723986 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:10.725876 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:10.726788 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:10.726799 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:10.727482 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:10.728263 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:10.728285 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.729856 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.729861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:10.731172 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:10.731186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.731980 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.731986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:10.732311 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:10.733008 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:10.733033 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:10.735431 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:10.738566 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:10.738577 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:10.738605 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:10.739084 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:10.739517 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:10.739522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:10.739525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:10.739746 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:10.740855 (+ 1109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:10.740865 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:10.741266 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 16:55:10.741555 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 16:55:10.742159 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":282,"threads_started":1}
I20260619 16:55:10.745162 17086 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36993
I20260619 16:55:10.745518 17086 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:10.746289 17086 heartbeater.cc:511] Master 127.16.48.62:36993 requested a full tablet report, sending...
I20260619 16:55:10.747903 16628 ts_manager.cc:194] Registered new tserver with Master: ab30578100f444ceb2bd58d285434d08 (127.16.48.3:39095)
I20260619 16:55:10.748821 16628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:37699
I20260619 16:55:10.757414 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:55:10 dist-test-slave-3ffd krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:55:10 dist-test-slave-3ffd krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:10.806211 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:10.788847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.16.48.1:41413)
0619 16:55:10.789573 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:10.789580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:10.789609 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:55:10.789735 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:10.789742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:10.789976 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:10.790137 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:10.791608 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.792748 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.794269 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.794881 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.794991 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:10.799626 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:10.799644 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:10.799664 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:10.799710 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:10.802974 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.803851 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.803860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.803862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.804004 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.804506 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.804510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.804512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.804886 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:10.805308 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:10.805554 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 16:55:10.805783 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":373,"threads_started":1}
W20260619 16:55:10.807577 16733 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59720
Jun 19 16:55:10 dist-test-slave-3ffd krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:55:10 dist-test-slave-3ffd krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:10.854936 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:10.840469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.16.48.1:41413)
0619 16:55:10.840731 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:10.840737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:10.841009 (+ 272us) server_negotiation.cc:407] Connection header received
0619 16:55:10.841203 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:10.841207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:10.841281 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:10.841369 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:10.842682 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.843364 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.844724 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.845205 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.845335 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:10.849556 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:10.849568 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:10.849571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:10.849621 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:10.852497 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.853238 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.853242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.853244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.853372 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.853702 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.853705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.853707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.853955 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:10.854209 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:10.854499 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 16:55:10.854703 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 16:55:10.856731 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 16690
I20260619 16:55:10.870711 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:10.857863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.16.48.1:41413)
0619 16:55:10.858048 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:10.858052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:10.858422 (+ 370us) server_negotiation.cc:407] Connection header received
0619 16:55:10.858629 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:10.858634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:10.858698 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:10.858792 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:10.860010 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.860911 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.862179 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.862608 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.862719 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:10.865580 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:10.865586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:10.865589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:10.865627 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:10.868225 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.868971 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.868975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.868978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.869126 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.869455 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.869458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.869460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.869736 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:10.869920 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:10.870229 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 16:55:10.870402 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:55:10.871439 16813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:10.955247 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 16690
I20260619 16:55:10.973735 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 16824
Jun 19 16:55:10 dist-test-slave-3ffd krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:55:10.990542 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:10.974814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.16.48.2:37187)
0619 16:55:10.975301 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:10.975309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:10.975334 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:55:10.975401 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:10.975407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:10.975617 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:10.975768 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:10.977242 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.978230 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.979810 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:10.980454 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:10.980556 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:10.984957 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:10.984970 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:10.984986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:10.985028 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:10.987726 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.988473 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.988479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.988482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.988644 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:10.989022 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:10.989026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:10.989028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:10.989385 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:10.989635 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:10.990042 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 16:55:10.990250 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":257,"threads_started":1}
I20260619 16:55:10.992015 16947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:11.075248 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 16824
I20260619 16:55:11.093513 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 16958
Jun 19 16:55:11 dist-test-slave-3ffd krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:55:11.110368 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:11.094457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.16.48.3:39095)
0619 16:55:11.095115 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:11.095122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:11.095154 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:55:11.095217 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:11.095224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:11.095451 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:11.095609 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:11.096926 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:11.098028 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:11.099376 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:11.099981 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:11.100094 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:11.104598 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:11.104614 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:11.104631 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:11.104683 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:11.107554 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:11.108478 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:11.108486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:11.108489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:11.108627 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:11.108984 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:11.108987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:11.108989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:11.109357 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:11.109596 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:11.109843 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 16:55:11.110057 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":332,"threads_started":1}
I20260619 16:55:11.111754 17081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:11.196036 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 16958
I20260619 16:55:11.211871 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 16596
I20260619 16:55:11.213249 16658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:11.315011 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 16596
2026-06-19T16:55:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4201 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:11 dist-test-slave-3ffd krb5kdc[17108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:11 dist-test-slave-3ffd krb5kdc[17108](info): set up 2 sockets
Jun 19 16:55:11 dist-test-slave-3ffd krb5kdc[17108](info): commencing operation
krb5kdc: starting...
W20260619 16:55:12.404510 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:12 dist-test-slave-3ffd krb5kdc[17108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:12.597124 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46679
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:12.938824 17121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:12.939232 17121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:12.939405 17121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:12.946998 17121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:12.947201 17121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:12.947315 17121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:12.947413 17121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:12.947494 17121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:12.957947 17121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:12.959398 17121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:12.960887 17121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:12.974104 17126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:12.974156 17129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:12.974074 17127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:12.975255 17121 server_base.cc:1034] running on GCE node
I20260619 16:55:12.976569 17121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:12.978272 17121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:12.979588 17121 hybrid_clock.cc:648] HybridClock initialized: now 1781888112979502 us; error 82 us; skew 500 ppm
I20260619 16:55:12.980017 17121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:12 dist-test-slave-3ffd krb5kdc[17108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888112, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:12.983462 17121 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:12.985327 17121 webserver.cc:460] Webserver started at http://127.16.48.62:39929/ using document root <none> and password file <none>
I20260619 16:55:12.986188 17121 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:12.986372 17121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:12.986802 17121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:12.990149 17121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "5b86b28f52584219a73b02d1ba0d46a0"
format_stamp: "Formatted at 2026-06-19 16:55:12 on dist-test-slave-3ffd"
server_key: "1d0f1f3c05785e531777a6323107890d"
server_key_iv: "37f8c69aa321bc6f298dcd23a314caa2"
server_key_version: "encryptionkey@0"
I20260619 16:55:12.991025 17121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "5b86b28f52584219a73b02d1ba0d46a0"
format_stamp: "Formatted at 2026-06-19 16:55:12 on dist-test-slave-3ffd"
server_key: "1d0f1f3c05785e531777a6323107890d"
server_key_iv: "37f8c69aa321bc6f298dcd23a314caa2"
server_key_version: "encryptionkey@0"
I20260619 16:55:12.996858 17121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:55:13.000991 17136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:13.002697 17121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:13.002974 17121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "5b86b28f52584219a73b02d1ba0d46a0"
format_stamp: "Formatted at 2026-06-19 16:55:12 on dist-test-slave-3ffd"
server_key: "1d0f1f3c05785e531777a6323107890d"
server_key_iv: "37f8c69aa321bc6f298dcd23a314caa2"
server_key_version: "encryptionkey@0"
I20260619 16:55:13.003213 17121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:13.021579 17121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:13.028841 17121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:13.029191 17121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:13.053299 17121 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:46679
I20260619 16:55:13.053299 17187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:46679 every 8 connection(s)
I20260619 16:55:13.055724 17121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:13.056318 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 17121
I20260619 16:55:13.056622 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:13.057483 16576 external_mini_cluster.cc:1428] Setting key 372535162f5274793d5d8c181b2da327
I20260619 16:55:13.061461 17188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:55:13 dist-test-slave-3ffd krb5kdc[17108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:13.079316 17188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0: Bootstrap starting.
I20260619 16:55:13.082710 17188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:13.083948 17188 log.cc:824] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:13.085204 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:13.059828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.16.48.62:46679)
0619 16:55:13.060687 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:13.060700 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:13.060748 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:55:13.062543 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:13.062568 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:13.063124 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:13.063564 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:13.065607 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:13.067860 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:13.069835 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:13.071073 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:13.071295 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:13.076839 (+ 5544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:13.076853 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:13.076877 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:13.076938 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:13.080329 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:13.081100 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:13.081108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:13.081118 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:13.081297 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:13.081711 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:13.081728 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:13.081731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:13.082406 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:13.082765 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:13.083296 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 16:55:13.083750 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":410,"threads_started":1}
I20260619 16:55:13.087649 17188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0: No bootstrap required, opened a new log
I20260619 16:55:13.095746 17188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b86b28f52584219a73b02d1ba0d46a0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46679 } }
I20260619 16:55:13.096273 17188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:13.096401 17188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b86b28f52584219a73b02d1ba0d46a0, State: Initialized, Role: FOLLOWER
I20260619 16:55:13.097084 17188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b86b28f52584219a73b02d1ba0d46a0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46679 } }
I20260619 16:55:13.097308 17188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:13.097471 17188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:13.097666 17188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:13.099920 17188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b86b28f52584219a73b02d1ba0d46a0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46679 } }
I20260619 16:55:13.100644 17188 leader_election.cc:302] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b86b28f52584219a73b02d1ba0d46a0; no voters:
I20260619 16:55:13.101384 17188 leader_election.cc:288] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:13.101528 17193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:13.102414 17193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [term 1 LEADER]: Becoming Leader. State: Replica: 5b86b28f52584219a73b02d1ba0d46a0, State: Running, Role: LEADER
I20260619 16:55:13.103026 17193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b86b28f52584219a73b02d1ba0d46a0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46679 } }
I20260619 16:55:13.103958 17188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:13.106936 17194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b86b28f52584219a73b02d1ba0d46a0. Latest consensus state: current_term: 1 leader_uuid: "5b86b28f52584219a73b02d1ba0d46a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b86b28f52584219a73b02d1ba0d46a0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46679 } } }
I20260619 16:55:13.106979 17195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b86b28f52584219a73b02d1ba0d46a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b86b28f52584219a73b02d1ba0d46a0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46679 } } }
I20260619 16:55:13.107586 17195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:13.107586 17194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:13.111673 17201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:13.117245 17201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:13.127597 17201 catalog_manager.cc:1347] Generated new cluster ID: 054f4732de1a46d0b5c0792929b5a40c
I20260619 16:55:13.127739 17201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:13.144040 17201 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:13.145553 17201 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:13.167989 17201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b86b28f52584219a73b02d1ba0d46a0: Generated new TSK 0
I20260619 16:55:13.169236 17201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:55:13.233909 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46679
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:13.617132 17214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:13.617494 17214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:13.617668 17214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:13.625622 17214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:13.625818 17214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:13.625994 17214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:13.636315 17214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46679
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:13.637701 17214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:13.639111 17214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:13.653097 17222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:13.653044 17219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:13.653168 17220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:13.653913 17214 server_base.cc:1034] running on GCE node
I20260619 16:55:13.654559 17214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:13.655596 17214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:13.656805 17214 hybrid_clock.cc:648] HybridClock initialized: now 1781888113656761 us; error 37 us; skew 500 ppm
I20260619 16:55:13.657157 17214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:13 dist-test-slave-3ffd krb5kdc[17108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888113, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:13.660394 17214 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:55:13.661974 17214 webserver.cc:460] Webserver started at http://127.16.48.1:39641/ using document root <none> and password file <none>
I20260619 16:55:13.662743 17214 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:13.662896 17214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:13.663234 17214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:13.666455 17214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "789e3834d53e431c9067637edacbc6ee"
format_stamp: "Formatted at 2026-06-19 16:55:13 on dist-test-slave-3ffd"
server_key: "a3c3008184221fd44b53b9fb10a2c8ee"
server_key_iv: "12ddc0de3ec1d1664d515e2ae04bb9b3"
server_key_version: "encryptionkey@0"
I20260619 16:55:13.667302 17214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "789e3834d53e431c9067637edacbc6ee"
format_stamp: "Formatted at 2026-06-19 16:55:13 on dist-test-slave-3ffd"
server_key: "a3c3008184221fd44b53b9fb10a2c8ee"
server_key_iv: "12ddc0de3ec1d1664d515e2ae04bb9b3"
server_key_version: "encryptionkey@0"
I20260619 16:55:13.673265 17214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 16:55:13.677289 17229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:13.678929 17214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:55:13.679157 17214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "789e3834d53e431c9067637edacbc6ee"
format_stamp: "Formatted at 2026-06-19 16:55:13 on dist-test-slave-3ffd"
server_key: "a3c3008184221fd44b53b9fb10a2c8ee"
server_key_iv: "12ddc0de3ec1d1664d515e2ae04bb9b3"
server_key_version: "encryptionkey@0"
I20260619 16:55:13.679410 17214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:13.700544 17214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:13.705812 17214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:13.706146 17214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:13.707311 17214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:13.709120 17214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:13.709224 17214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:13.709375 17214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:13.709444 17214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:13.744145 17214 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:46325
I20260619 16:55:13.744244 17341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:46325 every 8 connection(s)
I20260619 16:55:13.745616 17214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:13.746731 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 17214
I20260619 16:55:13.746969 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:13.747803 16576 external_mini_cluster.cc:1428] Setting key 89e92aabae0835fe617993d13a88e2c4
Jun 19 16:55:13 dist-test-slave-3ffd krb5kdc[17108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888113, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:55:13.771636 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:13.748970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:37501 (local address 127.16.48.62:46679)
0619 16:55:13.749512 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:13.749519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:13.750932 (+ 1413us) server_negotiation.cc:407] Connection header received
0619 16:55:13.754454 (+ 3522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:13.754459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:13.754547 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:13.754663 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:13.756975 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:13.757754 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:13.759432 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:13.759835 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:13.759990 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:13.764736 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:13.764770 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:13.764776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:13.764823 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:13.767577 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:13.768628 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:13.768633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:13.768637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:13.768766 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:13.769452 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:13.769457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:13.769459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:13.769949 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:13.770109 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:13.771166 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0619 16:55:13.771371 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":277,"threads_started":1}
I20260619 16:55:13.773407 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:13.749710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46679 (local address 127.16.48.1:37501)
0619 16:55:13.750731 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:13.750786 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:13.753899 (+ 3113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:13.755012 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:13.755029 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:13.755955 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:13.756757 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:13.756778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:13.757947 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:13.757952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:13.759258 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:13.759271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:13.760099 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:13.760106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:13.760411 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:13.761155 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:13.761181 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:13.764402 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:13.767767 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:13.767778 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:13.767800 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:13.768479 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:13.768918 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:13.768926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:13.768932 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:13.769250 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:13.770118 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:13.770130 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:13.770668 (+ 538us) client_negotiation.cc:769] Sending connection context
0619 16:55:13.771030 (+ 362us) client_negotiation.cc:240] Negotiation successful
0619 16:55:13.771698 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":399,"threads_started":1}
I20260619 16:55:13.775327 17342 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46679
I20260619 16:55:13.775875 17342 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:13.776800 17342 heartbeater.cc:511] Master 127.16.48.62:46679 requested a full tablet report, sending...
I20260619 16:55:13.779074 17153 ts_manager.cc:194] Registered new tserver with Master: 789e3834d53e431c9067637edacbc6ee (127.16.48.1:46325)
I20260619 16:55:13.780701 17153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:37501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:55:13.795470 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46679
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:14.139845 17348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:14.140251 17348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:14.140442 17348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:14.148077 17348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:14.148343 17348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:14.148540 17348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:14.159381 17348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46679
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:14.160894 17348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:14.162324 17348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:14.176816 17353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:14.176981 17354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:14.177263 17356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:14.178416 17348 server_base.cc:1034] running on GCE node
I20260619 16:55:14.179224 17348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:14.180341 17348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:14.181556 17348 hybrid_clock.cc:648] HybridClock initialized: now 1781888114181505 us; error 44 us; skew 500 ppm
I20260619 16:55:14.181917 17348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:14 dist-test-slave-3ffd krb5kdc[17108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888114, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:14.191371 17348 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:55:14.193440 17348 webserver.cc:460] Webserver started at http://127.16.48.2:34823/ using document root <none> and password file <none>
I20260619 16:55:14.194321 17348 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:14.194506 17348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:14.194878 17348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:14.198289 17348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f"
format_stamp: "Formatted at 2026-06-19 16:55:14 on dist-test-slave-3ffd"
server_key: "d735b761323b4ef6422c0c2c8c8e0bb6"
server_key_iv: "59ceb23bab25987d960faa7f82c69ffc"
server_key_version: "encryptionkey@0"
I20260619 16:55:14.199291 17348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f"
format_stamp: "Formatted at 2026-06-19 16:55:14 on dist-test-slave-3ffd"
server_key: "d735b761323b4ef6422c0c2c8c8e0bb6"
server_key_iv: "59ceb23bab25987d960faa7f82c69ffc"
server_key_version: "encryptionkey@0"
I20260619 16:55:14.205921 17348 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260619 16:55:14.210984 17363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:14.212637 17348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:14.212877 17348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f"
format_stamp: "Formatted at 2026-06-19 16:55:14 on dist-test-slave-3ffd"
server_key: "d735b761323b4ef6422c0c2c8c8e0bb6"
server_key_iv: "59ceb23bab25987d960faa7f82c69ffc"
server_key_version: "encryptionkey@0"
I20260619 16:55:14.213105 17348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:14.232638 17348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:14.237596 17348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:14.237931 17348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:14.238977 17348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:14.240566 17348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:14.240675 17348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:14.240849 17348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:14.240927 17348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:14.276487 17348 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:35497
I20260619 16:55:14.276535 17475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:35497 every 8 connection(s)
I20260619 16:55:14.277911 17348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:14.285454 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 17348
I20260619 16:55:14.285668 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:14.286109 16576 external_mini_cluster.cc:1428] Setting key fd1f9d4b181164dc68062606a6a4219c
Jun 19 16:55:14 dist-test-slave-3ffd krb5kdc[17108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888114, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:14.301007 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.280923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:47343 (local address 127.16.48.62:46679)
0619 16:55:14.281373 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:14.281378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:14.282378 (+ 1000us) server_negotiation.cc:407] Connection header received
0619 16:55:14.284384 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:14.284389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:14.284489 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:14.284612 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:14.286451 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.287222 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.289064 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.289916 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.290082 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.294520 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:14.294526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:14.294529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:14.294566 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:14.297462 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:14.298403 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:14.298414 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:14.298417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:14.298566 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:14.299061 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:14.299068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:14.299076 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:14.299357 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:14.299523 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:14.300365 (+ 842us) server_negotiation.cc:299] Negotiation successful
0619 16:55:14.300626 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
I20260619 16:55:14.302101 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.281439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46679 (local address 127.16.48.2:47343)
0619 16:55:14.282192 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:14.282240 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:14.284012 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:14.284885 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:14.284895 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:14.285513 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:14.286268 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.286290 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.287398 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.287402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.288895 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.288908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.290205 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.290211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.290458 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.291317 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:14.291348 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:14.294137 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:14.297707 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:14.297719 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:14.297746 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:14.298241 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:14.298696 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:14.298702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:14.298706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:14.298928 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:14.299501 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:14.299510 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:14.299901 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 16:55:14.300268 (+ 367us) client_negotiation.cc:240] Negotiation successful
0619 16:55:14.300873 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":286,"threads_started":1}
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
I20260619 16:55:14.303987 17476 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46679
I20260619 16:55:14.304400 17476 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:14.305346 17476 heartbeater.cc:511] Master 127.16.48.62:46679 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
I20260619 16:55:14.307287 17153 ts_manager.cc:194] Registered new tserver with Master: fc5f5a720da84a8a88fa1af35f7d7d7f (127.16.48.2:35497)
I20260619 16:55:14.308337 17153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:47343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:55:14.331259 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46679
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:14.690193 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:14.690539 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:14.690649 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:14.700814 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:14.701067 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:14.701288 17482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:14.713961 17482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46679
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:14.715356 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:14.716952 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:14.731129 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:14.731114 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:14.731194 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:14.732018 17482 server_base.cc:1034] running on GCE node
I20260619 16:55:14.732761 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:14.733845 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:14.735092 17482 hybrid_clock.cc:648] HybridClock initialized: now 1781888114735031 us; error 59 us; skew 500 ppm
I20260619 16:55:14.735432 17482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:14 dist-test-slave-3ffd krb5kdc[17108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888114, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:14.738776 17482 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:55:14.740545 17482 webserver.cc:460] Webserver started at http://127.16.48.3:42989/ using document root <none> and password file <none>
I20260619 16:55:14.741313 17482 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:14.741496 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:14.741848 17482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:14.745039 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "6cd03cd5776f4a28b711233c97a0474d"
format_stamp: "Formatted at 2026-06-19 16:55:14 on dist-test-slave-3ffd"
server_key: "38293d42896b179e2729531b7a3217f2"
server_key_iv: "b49cb2062e90f7daa37fc48d8018c27a"
server_key_version: "encryptionkey@0"
I20260619 16:55:14.745919 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "6cd03cd5776f4a28b711233c97a0474d"
format_stamp: "Formatted at 2026-06-19 16:55:14 on dist-test-slave-3ffd"
server_key: "38293d42896b179e2729531b7a3217f2"
server_key_iv: "b49cb2062e90f7daa37fc48d8018c27a"
server_key_version: "encryptionkey@0"
I20260619 16:55:14.751499 17482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:55:14.755514 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:14.757153 17482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:14.757357 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "6cd03cd5776f4a28b711233c97a0474d"
format_stamp: "Formatted at 2026-06-19 16:55:14 on dist-test-slave-3ffd"
server_key: "38293d42896b179e2729531b7a3217f2"
server_key_iv: "b49cb2062e90f7daa37fc48d8018c27a"
server_key_version: "encryptionkey@0"
I20260619 16:55:14.757622 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:14.784132 17342 heartbeater.cc:503] Master 127.16.48.62:46679 was elected leader, sending a full tablet report...
I20260619 16:55:14.797331 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:14.801748 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:14.802014 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:14.803013 17482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:14.804553 17482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:14.804641 17482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:14.804728 17482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:14.804819 17482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:14.839008 17482 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:35239
I20260619 16:55:14.839068 17609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:35239 every 8 connection(s)
I20260619 16:55:14.840440 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:14.841342 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 17482
I20260619 16:55:14.841542 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:14.842044 16576 external_mini_cluster.cc:1428] Setting key 12031768a3413db40d03793150183dd8
Jun 19 16:55:14 dist-test-slave-3ffd krb5kdc[17108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888114, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:14.861740 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.843433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:50293 (local address 127.16.48.62:46679)
0619 16:55:14.843945 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:14.843951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:14.845369 (+ 1418us) server_negotiation.cc:407] Connection header received
0619 16:55:14.847119 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:14.847123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:14.847200 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:14.847294 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:14.848952 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.849759 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.851397 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.851796 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.851889 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.855350 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:14.855355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:14.855358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:14.855392 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:14.858793 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:14.859544 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:14.859548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:14.859550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:14.859666 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:14.860049 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:14.860052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:14.860053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:14.860370 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:14.860477 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:14.861233 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 16:55:14.861401 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
I20260619 16:55:14.862783 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.844231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46679 (local address 127.16.48.3:50293)
0619 16:55:14.845057 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:14.845114 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:14.846857 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:14.847510 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:14.847519 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:14.848086 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:14.848780 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.848799 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.849918 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.849922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.851257 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.851269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.851990 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.851994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.852271 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.852923 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:14.852946 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:14.855119 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:14.858933 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:14.858940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:14.858955 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:14.859425 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:14.859760 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:14.859764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:14.859766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:14.859951 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:14.860472 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:14.860479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:14.860830 (+ 351us) client_negotiation.cc:769] Sending connection context
0619 16:55:14.861082 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 16:55:14.861488 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":344,"threads_started":1}
I20260619 16:55:14.864300 17610 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46679
I20260619 16:55:14.864607 17610 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:14.865393 17610 heartbeater.cc:511] Master 127.16.48.62:46679 requested a full tablet report, sending...
I20260619 16:55:14.866974 17153 ts_manager.cc:194] Registered new tserver with Master: 6cd03cd5776f4a28b711233c97a0474d (127.16.48.3:35239)
I20260619 16:55:14.867800 17153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:50293
I20260619 16:55:14.870694 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:55:14.891459 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.877347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.16.48.62:46679)
0619 16:55:14.877524 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:14.877529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:14.878531 (+ 1002us) server_negotiation.cc:407] Connection header received
0619 16:55:14.878696 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:14.878700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:14.878802 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:14.878899 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:14.881067 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.881820 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.883294 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.883697 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.883810 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.886247 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:14.886253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:14.886256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:14.886297 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:14.889064 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:14.889774 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:14.889778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:14.889780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:14.889897 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:14.890255 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:14.890258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:14.890260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:14.890574 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:14.890748 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:14.891039 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 16:55:14.891229 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:55:14.898558 17153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:55:14.906387 17153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:14.943569 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.919890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.16.48.3:35239)
0619 16:55:14.933833 (+ 13943us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:14.933846 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:14.933894 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:55:14.934046 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:14.934060 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:14.934456 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:14.934784 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:14.936014 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.937837 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.940784 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.942311 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.942459 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.942721 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:14.943009 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 16:55:14.943177 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13713,"thread_start_us":13549,"threads_started":1}
I20260619 16:55:14.943616 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.919667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:35239 (local address 127.0.0.1:34496)
0619 16:55:14.929235 (+ 9568us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:14.929307 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:14.929570 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:14.935241 (+ 5671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:14.935254 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:14.935318 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:14.935728 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.935747 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.937995 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.937999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.940564 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.940582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.942517 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.942523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.942738 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.942801 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 16:55:14.942961 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 16:55:14.943177 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9184,"mutex_wait_us":2,"thread_start_us":16611,"threads_started":1}
I20260619 16:55:14.945019 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.919762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:46325 (local address 127.0.0.1:52464)
0619 16:55:14.936093 (+ 16331us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:14.936139 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:14.936394 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:14.937194 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:14.937198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:14.937222 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:14.937618 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.937627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.939575 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.939581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.941957 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.941970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.944215 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.944225 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.944399 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.944429 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:55:14.944522 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:55:14.944647 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16195,"mutex_wait_us":47,"thread_start_us":8891,"threads_started":1}
I20260619 16:55:14.945076 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.920416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.16.48.1:46325)
0619 16:55:14.932500 (+ 12084us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:14.932515 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:14.936259 (+ 3744us) server_negotiation.cc:407] Connection header received
0619 16:55:14.936658 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:14.936669 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:14.936948 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:14.937209 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:14.937801 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.939403 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.942145 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.943971 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.944103 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.944368 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:14.944587 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 16:55:14.944758 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11826,"thread_start_us":12110,"threads_started":1}
I20260619 16:55:14.946276 17627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.930216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.16.48.2:35497)
0619 16:55:14.935656 (+ 5440us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:14.935664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:14.936515 (+ 851us) server_negotiation.cc:407] Connection header received
0619 16:55:14.936826 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:14.936834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:14.937082 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:14.937287 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:14.938125 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.939149 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.941703 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.943573 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.943726 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.943932 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:14.945778 (+ 1846us) server_negotiation.cc:299] Negotiation successful
0619 16:55:14.945958 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5278,"thread_start_us":5160,"threads_started":1}
I20260619 16:55:14.946707 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:14.933969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:35497 (local address 127.0.0.1:45748)
0619 16:55:14.936401 (+ 2432us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:14.936419 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:14.936619 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:14.937510 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:14.937515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:14.937540 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:14.937952 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.937964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.939295 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.939302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.941517 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:14.941529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:14.944072 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:14.944078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:14.944300 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:14.944341 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:55:14.944430 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:55:14.944600 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2298,"mutex_wait_us":85,"thread_start_us":1804,"threads_started":1}
I20260619 16:55:14.948505 17545 tablet_service.cc:1467] Processing CreateTablet for tablet 9c733cf61b714df1bcb94f9d274d5a4c (DEFAULT_TABLE table=test-table [id=0dd34a021cdb494f80636d52bda7047d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:14.948782 17277 tablet_service.cc:1467] Processing CreateTablet for tablet 9c733cf61b714df1bcb94f9d274d5a4c (DEFAULT_TABLE table=test-table [id=0dd34a021cdb494f80636d52bda7047d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:14.949712 17411 tablet_service.cc:1467] Processing CreateTablet for tablet 9c733cf61b714df1bcb94f9d274d5a4c (DEFAULT_TABLE table=test-table [id=0dd34a021cdb494f80636d52bda7047d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:14.950147 17545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c733cf61b714df1bcb94f9d274d5a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:14.950147 17277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c733cf61b714df1bcb94f9d274d5a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:14.950786 17411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c733cf61b714df1bcb94f9d274d5a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:14.963831 17629 tablet_bootstrap.cc:492] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f: Bootstrap starting.
I20260619 16:55:14.965905 17630 tablet_bootstrap.cc:492] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee: Bootstrap starting.
I20260619 16:55:14.966730 17631 tablet_bootstrap.cc:492] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d: Bootstrap starting.
I20260619 16:55:14.968137 17629 tablet_bootstrap.cc:654] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:14.969604 17629 log.cc:824] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:14.971067 17631 tablet_bootstrap.cc:654] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:14.971619 17630 tablet_bootstrap.cc:654] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:14.972534 17631 log.cc:824] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:14.973023 17630 log.cc:824] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:14.973788 17629 tablet_bootstrap.cc:492] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f: No bootstrap required, opened a new log
I20260619 16:55:14.974174 17629 ts_tablet_manager.cc:1397] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 16:55:14.975906 17630 tablet_bootstrap.cc:492] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee: No bootstrap required, opened a new log
I20260619 16:55:14.976238 17630 ts_tablet_manager.cc:1397] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260619 16:55:14.982887 17631 tablet_bootstrap.cc:492] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d: No bootstrap required, opened a new log
I20260619 16:55:14.983227 17631 ts_tablet_manager.cc:1397] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260619 16:55:14.984834 17630 raft_consensus.cc:348] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } }
I20260619 16:55:14.985230 17630 raft_consensus.cc:374] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:14.985342 17630 raft_consensus.cc:729] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789e3834d53e431c9067637edacbc6ee, State: Initialized, Role: FOLLOWER
I20260619 16:55:14.986027 17630 consensus_queue.cc:260] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } }
I20260619 16:55:14.986873 17629 raft_consensus.cc:348] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } }
I20260619 16:55:14.987725 17629 raft_consensus.cc:374] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:14.987880 17629 raft_consensus.cc:729] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5f5a720da84a8a88fa1af35f7d7d7f, State: Initialized, Role: FOLLOWER
I20260619 16:55:14.987910 17630 ts_tablet_manager.cc:1428] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260619 16:55:14.988621 17629 consensus_queue.cc:260] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } }
I20260619 16:55:14.989773 17476 heartbeater.cc:503] Master 127.16.48.62:46679 was elected leader, sending a full tablet report...
I20260619 16:55:14.990483 17629 ts_tablet_manager.cc:1428] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260619 16:55:14.994313 17631 raft_consensus.cc:348] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } }
I20260619 16:55:14.994740 17631 raft_consensus.cc:374] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:14.994848 17631 raft_consensus.cc:729] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cd03cd5776f4a28b711233c97a0474d, State: Initialized, Role: FOLLOWER
I20260619 16:55:14.995443 17631 consensus_queue.cc:260] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } }
I20260619 16:55:14.996743 17610 heartbeater.cc:503] Master 127.16.48.62:46679 was elected leader, sending a full tablet report...
I20260619 16:55:14.997395 17631 ts_tablet_manager.cc:1428] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260619 16:55:14.998389 17343 tablet.cc:2369] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:55:15.030133 17477 tablet.cc:2369] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:55:15.056356 17635 raft_consensus.cc:482] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:15.056667 17635 raft_consensus.cc:504] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } }
I20260619 16:55:15.058940 17635 leader_election.cc:288] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cd03cd5776f4a28b711233c97a0474d (127.16.48.3:35239), fc5f5a720da84a8a88fa1af35f7d7d7f (127.16.48.2:35497)
I20260619 16:55:15.067560 17627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.060005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:43321 (local address 127.16.48.2:35497)
0619 16:55:15.060275 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:15.060283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:15.060973 (+ 690us) server_negotiation.cc:407] Connection header received
0619 16:55:15.061283 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:15.061289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:15.061385 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:15.061495 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:15.062255 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.063443 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.065523 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.066494 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.066640 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.066739 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:15.067093 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 16:55:15.067253 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:55:15.068094 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.059291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:35497 (local address 127.16.48.1:43321)
0619 16:55:15.060841 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:15.060866 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:15.061098 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:15.061524 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:15.061601 (+ 77us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:15.061639 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:15.062087 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:15.062097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.063645 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.063649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:15.065340 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:15.065354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.066727 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.066733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:15.066900 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.066940 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:55:15.067043 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:55:15.067414 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"mutex_wait_us":9,"thread_start_us":1524,"threads_started":1}
I20260619 16:55:15.068584 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.059337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:35239 (local address 127.16.48.1:45331)
0619 16:55:15.061159 (+ 1822us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:15.061182 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:15.061291 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:15.061678 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:15.061683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:15.061713 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:15.062172 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:15.062184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.063600 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.063606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:15.065962 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:15.065977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.067684 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.067693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:15.067885 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.067911 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:55:15.068027 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 16:55:15.068196 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1697,"mutex_wait_us":21,"thread_start_us":1137,"threads_started":1}
I20260619 16:55:15.068463 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.059417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:45331 (local address 127.16.48.3:35239)
0619 16:55:15.059678 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:15.059688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:15.061259 (+ 1571us) server_negotiation.cc:407] Connection header received
0619 16:55:15.061442 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:15.061447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:15.061546 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:15.061631 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:15.062361 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.063453 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.066147 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.067279 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.067405 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.067508 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:15.068058 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 16:55:15.068221 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:55:15.069389 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c733cf61b714df1bcb94f9d274d5a4c" candidate_uuid: "789e3834d53e431c9067637edacbc6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" is_pre_election: true
I20260619 16:55:15.070144 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c733cf61b714df1bcb94f9d274d5a4c" candidate_uuid: "789e3834d53e431c9067637edacbc6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cd03cd5776f4a28b711233c97a0474d" is_pre_election: true
I20260619 16:55:15.070303 17431 raft_consensus.cc:2454] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789e3834d53e431c9067637edacbc6ee in term 0.
I20260619 16:55:15.070639 17565 raft_consensus.cc:2454] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789e3834d53e431c9067637edacbc6ee in term 0.
I20260619 16:55:15.071094 17231 leader_election.cc:302] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789e3834d53e431c9067637edacbc6ee, fc5f5a720da84a8a88fa1af35f7d7d7f; no voters:
I20260619 16:55:15.071532 17635 raft_consensus.cc:2781] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:15.071708 17635 raft_consensus.cc:482] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:15.071807 17635 raft_consensus.cc:3037] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:15.074232 17635 raft_consensus.cc:504] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } }
I20260619 16:55:15.074960 17635 leader_election.cc:288] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [CANDIDATE]: Term 1 election: Requested vote from peers 6cd03cd5776f4a28b711233c97a0474d (127.16.48.3:35239), fc5f5a720da84a8a88fa1af35f7d7d7f (127.16.48.2:35497)
I20260619 16:55:15.075532 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c733cf61b714df1bcb94f9d274d5a4c" candidate_uuid: "789e3834d53e431c9067637edacbc6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cd03cd5776f4a28b711233c97a0474d"
I20260619 16:55:15.075630 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c733cf61b714df1bcb94f9d274d5a4c" candidate_uuid: "789e3834d53e431c9067637edacbc6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f"
I20260619 16:55:15.075759 17565 raft_consensus.cc:3037] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:15.075906 17431 raft_consensus.cc:3037] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:15.078132 17565 raft_consensus.cc:2454] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789e3834d53e431c9067637edacbc6ee in term 1.
I20260619 16:55:15.078137 17431 raft_consensus.cc:2454] T 9c733cf61b714df1bcb94f9d274d5a4c P fc5f5a720da84a8a88fa1af35f7d7d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789e3834d53e431c9067637edacbc6ee in term 1.
I20260619 16:55:15.078652 17233 leader_election.cc:302] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cd03cd5776f4a28b711233c97a0474d, 789e3834d53e431c9067637edacbc6ee; no voters:
I20260619 16:55:15.078992 17635 raft_consensus.cc:2781] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:15.079487 17635 raft_consensus.cc:686] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [term 1 LEADER]: Becoming Leader. State: Replica: 789e3834d53e431c9067637edacbc6ee, State: Running, Role: LEADER
I20260619 16:55:15.080053 17635 consensus_queue.cc:237] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } }
I20260619 16:55:15.085100 17152 catalog_manager.cc:5495] T 9c733cf61b714df1bcb94f9d274d5a4c P 789e3834d53e431c9067637edacbc6ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 789e3834d53e431c9067637edacbc6ee (127.16.48.1). New cstate: current_term: 1 leader_uuid: "789e3834d53e431c9067637edacbc6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd03cd5776f4a28b711233c97a0474d" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789e3834d53e431c9067637edacbc6ee" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5f5a720da84a8a88fa1af35f7d7d7f" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35497 } health_report { overall_health: UNKNOWN } } }
W20260619 16:55:15.092283 17611 tablet.cc:2369] T 9c733cf61b714df1bcb94f9d274d5a4c P 6cd03cd5776f4a28b711233c97a0474d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:15.138598 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.122849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.16.48.1:46325)
0619 16:55:15.123103 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:15.123110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:15.123544 (+ 434us) server_negotiation.cc:407] Connection header received
0619 16:55:15.123756 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:15.123766 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:15.123849 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:15.123961 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:15.125553 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.126270 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.127572 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.127998 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.128170 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.132457 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:15.132474 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:15.132492 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:15.132532 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:15.135739 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.136581 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.136589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.136591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.136781 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.137142 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.137145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.137147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.137491 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:15.137733 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:15.138005 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 16:55:15.138214 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888115, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888115, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:15.187485 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.172433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.16.48.1:46325)
0619 16:55:15.172650 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:15.172658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:15.173160 (+ 502us) server_negotiation.cc:407] Connection header received
0619 16:55:15.173379 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:15.173385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:15.173465 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:15.173578 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:15.175020 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.175768 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.177170 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.177621 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.177777 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.181983 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:15.181989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:15.181993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:15.182037 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:15.184819 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.185540 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.185544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.185546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.185674 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.186048 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.186051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.186053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.186320 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:15.186943 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:15.187046 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 16:55:15.187234 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:15.236028 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.221674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.16.48.1:46325)
0619 16:55:15.221919 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:15.221929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:15.222281 (+ 352us) server_negotiation.cc:407] Connection header received
0619 16:55:15.222424 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:15.222427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:15.222510 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:15.222601 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:15.223973 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.224821 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.226054 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.226486 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.226588 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.230672 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:15.230678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:15.230681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:15.230720 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:15.233415 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.234203 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.234208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.234210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.234333 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.234775 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.234778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.234780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.235039 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:15.235247 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:15.235488 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 16:55:15.235662 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 16:55:15.237872 17257 tablet_service.cc:2908] Rejecting scan request for tablet 9c733cf61b714df1bcb94f9d274d5a4c: Uninitialized: clean time has not yet been initialized
I20260619 16:55:15.241073 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 17214
I20260619 16:55:15.254784 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.242046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.16.48.1:46325)
0619 16:55:15.242280 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:15.242287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:15.242564 (+ 277us) server_negotiation.cc:407] Connection header received
0619 16:55:15.242705 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:15.242709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:15.242772 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:15.242851 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:15.244482 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.245186 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.246344 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.246740 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.246834 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.249629 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:15.249634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:15.249637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:15.249670 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:15.252273 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.252979 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.252983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.252985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.253114 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.253450 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.253453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.253455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.253722 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:15.253906 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:15.254176 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 16:55:15.254374 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:55:15.255450 17337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:15.357378 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 17214
I20260619 16:55:15.375106 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 17348
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:55:15.390609 17627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.375988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.16.48.2:35497)
0619 16:55:15.376187 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:15.376195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:15.376227 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:55:15.376440 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:15.376452 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:15.376549 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:15.376668 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:15.377850 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.378632 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.379960 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.380518 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.380665 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.384879 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:15.384896 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:15.384913 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:15.384955 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:15.387780 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.388678 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.388686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.388688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.388827 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.389260 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.389264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.389266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.389629 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:15.389871 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:15.390161 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 16:55:15.390352 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:55:15.391355 17471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:15.480934 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 17348
I20260619 16:55:15.498307 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 17482
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:55:15.513605 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:15.499358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.16.48.3:35239)
0619 16:55:15.499549 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:15.499554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:15.499570 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:55:15.499622 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:15.499626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:15.499690 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:15.499813 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:15.501109 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.501826 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.503097 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:15.503498 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:15.503674 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:15.508230 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:15.508257 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:15.508276 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:15.508317 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:15.510962 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.511681 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.511688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.511690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.511816 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:15.512209 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:15.512212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:15.512214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:15.512556 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:15.512792 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:15.513033 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 16:55:15.513200 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:55:15.514446 17605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:15.603464 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 17482
I20260619 16:55:15.622404 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 17121
I20260619 16:55:15.623634 17183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:15.731081 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 17121
2026-06-19T16:55:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4420 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17659](info): set up 2 sockets
Jun 19 16:55:15 dist-test-slave-3ffd krb5kdc[17659](info): commencing operation
krb5kdc: starting...
W20260619 16:55:16.828583 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:16 dist-test-slave-3ffd krb5kdc[17659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:17.025226 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:40121
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:40121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:17.358480 17672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:17.358891 17672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:17.359061 17672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:17.366582 17672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:17.366772 17672 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:55:17.366873 17672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:17.366994 17672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:17.367089 17672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:17.367179 17672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:17.377640 17672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:40121
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:17.379068 17672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:17.380501 17672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:17.393146 17677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:17.393163 17678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:17.393389 17680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:17.394029 17672 server_base.cc:1034] running on GCE node
I20260619 16:55:17.394944 17672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:17.396668 17672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:17.397935 17672 hybrid_clock.cc:648] HybridClock initialized: now 1781888117397853 us; error 69 us; skew 500 ppm
I20260619 16:55:17.398317 17672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:17 dist-test-slave-3ffd krb5kdc[17659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888117, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:17.401965 17672 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:17.403571 17672 webserver.cc:460] Webserver started at http://127.16.48.62:44921/ using document root <none> and password file <none>
I20260619 16:55:17.404414 17672 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:17.404516 17672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:17.404806 17672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:17.407989 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "02951ec0faa94564bb6d617fe20b3aa2"
format_stamp: "Formatted at 2026-06-19 16:55:17 on dist-test-slave-3ffd"
server_key: "3364d46c2a535c10d482eda783583cc6"
server_key_iv: "3b8bd2a7d56cbcba6c053f2fc9129d1d"
server_key_version: "encryptionkey@0"
I20260619 16:55:17.408900 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "02951ec0faa94564bb6d617fe20b3aa2"
format_stamp: "Formatted at 2026-06-19 16:55:17 on dist-test-slave-3ffd"
server_key: "3364d46c2a535c10d482eda783583cc6"
server_key_iv: "3b8bd2a7d56cbcba6c053f2fc9129d1d"
server_key_version: "encryptionkey@0"
I20260619 16:55:17.414489 17672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:55:17.418190 17687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:17.419755 17672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:55:17.419925 17672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "02951ec0faa94564bb6d617fe20b3aa2"
format_stamp: "Formatted at 2026-06-19 16:55:17 on dist-test-slave-3ffd"
server_key: "3364d46c2a535c10d482eda783583cc6"
server_key_iv: "3b8bd2a7d56cbcba6c053f2fc9129d1d"
server_key_version: "encryptionkey@0"
I20260619 16:55:17.420205 17672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:17.434544 17672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:17.439321 17672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:17.439564 17672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:17.463938 17672 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:40121
I20260619 16:55:17.463944 17748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:40121 every 8 connection(s)
I20260619 16:55:17.465523 17672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:17.471020 17749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:17.474884 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 17672
I20260619 16:55:17.475163 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:17.475894 16576 external_mini_cluster.cc:1428] Setting key 194efe460079763afea8c78da97216ec
I20260619 16:55:17.484813 17749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2: Bootstrap starting.
I20260619 16:55:17.488235 17749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:17.489319 17749 log.cc:824] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:55:17 dist-test-slave-3ffd krb5kdc[17659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:17.492919 17749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2: No bootstrap required, opened a new log
I20260619 16:55:17.499956 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:17.478219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.16.48.62:40121)
0619 16:55:17.479144 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:17.479158 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:17.479221 (+ 63us) server_negotiation.cc:407] Connection header received
0619 16:55:17.481085 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:17.481112 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:17.481574 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:17.482066 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:17.483551 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:17.485038 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:17.486324 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:17.487025 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:17.487233 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:17.491672 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:17.491681 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:17.491700 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:17.491737 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:17.495399 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:17.496246 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:17.496254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:17.496265 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:17.496450 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:17.496871 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:17.496876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:17.496882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:17.497492 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:17.497829 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:17.498330 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 16:55:17.498808 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":460,"threads_started":1}
I20260619 16:55:17.501282 17749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02951ec0faa94564bb6d617fe20b3aa2" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40121 } }
I20260619 16:55:17.501719 17749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:17.501848 17749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02951ec0faa94564bb6d617fe20b3aa2, State: Initialized, Role: FOLLOWER
I20260619 16:55:17.502434 17749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02951ec0faa94564bb6d617fe20b3aa2" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40121 } }
I20260619 16:55:17.502708 17749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:17.502864 17749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:17.503072 17749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:17.505533 17749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02951ec0faa94564bb6d617fe20b3aa2" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40121 } }
I20260619 16:55:17.506206 17749 leader_election.cc:302] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02951ec0faa94564bb6d617fe20b3aa2; no voters:
I20260619 16:55:17.506909 17749 leader_election.cc:288] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:17.507021 17754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:17.507705 17754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 02951ec0faa94564bb6d617fe20b3aa2, State: Running, Role: LEADER
I20260619 16:55:17.508380 17754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02951ec0faa94564bb6d617fe20b3aa2" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40121 } }
I20260619 16:55:17.509410 17749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:17.511651 17755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02951ec0faa94564bb6d617fe20b3aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02951ec0faa94564bb6d617fe20b3aa2" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40121 } } }
I20260619 16:55:17.512110 17755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:17.514696 17756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02951ec0faa94564bb6d617fe20b3aa2. Latest consensus state: current_term: 1 leader_uuid: "02951ec0faa94564bb6d617fe20b3aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02951ec0faa94564bb6d617fe20b3aa2" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40121 } } }
I20260619 16:55:17.515115 17756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:17.516050 17760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:17.523792 17760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:17.535127 17760 catalog_manager.cc:1347] Generated new cluster ID: 49ca382d65ce43a18551784a881c7a24
I20260619 16:55:17.535290 17760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:17.572317 17760 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:17.574361 17760 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:17.591745 17760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2: Generated new TSK 0
I20260619 16:55:17.592864 17760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:55:17.644687 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:40121
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:17.998476 17775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:17.998910 17775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:17.999096 17775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:18.007164 17775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:18.007381 17775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:18.007534 17775 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:55:18.007671 17775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:18.018733 17775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.48.62:40121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:18.020290 17775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:18.021781 17775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:18.037017 17781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:18.037084 17783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:18.037118 17780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:18.037930 17775 server_base.cc:1034] running on GCE node
I20260619 16:55:18.038659 17775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:18.039788 17775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:18.041069 17775 hybrid_clock.cc:648] HybridClock initialized: now 1781888118040965 us; error 88 us; skew 500 ppm
I20260619 16:55:18.041407 17775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:18 dist-test-slave-3ffd krb5kdc[17659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888118, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:18.044927 17775 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:55:18.046548 17775 webserver.cc:460] Webserver started at http://127.16.48.1:37651/ using document root <none> and password file <none>
I20260619 16:55:18.047273 17775 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:18.047364 17775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:18.047662 17775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:18.050758 17775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "a2337128874c44b6ac7610cd44bf8f78"
format_stamp: "Formatted at 2026-06-19 16:55:18 on dist-test-slave-3ffd"
server_key: "f945488137fc4c2de2ef95279ae378b6"
server_key_iv: "eb69d818d1256332ec0493670093f047"
server_key_version: "encryptionkey@0"
I20260619 16:55:18.051584 17775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "a2337128874c44b6ac7610cd44bf8f78"
format_stamp: "Formatted at 2026-06-19 16:55:18 on dist-test-slave-3ffd"
server_key: "f945488137fc4c2de2ef95279ae378b6"
server_key_iv: "eb69d818d1256332ec0493670093f047"
server_key_version: "encryptionkey@0"
I20260619 16:55:18.057726 17775 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 16:55:18.061677 17790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:18.063282 17775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:18.063448 17775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "a2337128874c44b6ac7610cd44bf8f78"
format_stamp: "Formatted at 2026-06-19 16:55:18 on dist-test-slave-3ffd"
server_key: "f945488137fc4c2de2ef95279ae378b6"
server_key_iv: "eb69d818d1256332ec0493670093f047"
server_key_version: "encryptionkey@0"
I20260619 16:55:18.063663 17775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:18.085549 17775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:18.090039 17775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:18.090289 17775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:18.093436 17775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:18.093530 17775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:18.093719 17775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:18.093776 17775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:55:18 dist-test-slave-3ffd krb5kdc[17659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888118, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:18.124010 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:18.094081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:35131 (local address 127.16.48.62:40121)
0619 16:55:18.094744 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:18.094751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:18.096095 (+ 1344us) server_negotiation.cc:407] Connection header received
0619 16:55:18.099483 (+ 3388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:18.099491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:18.099609 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:18.099762 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:18.102908 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.103978 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.107737 (+ 3759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.108601 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.108758 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:18.115688 (+ 6930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:18.115712 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:18.115716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:18.115757 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:18.119356 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:18.120839 (+ 1483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:18.120844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:18.120847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:18.120982 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:18.121723 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:18.121727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:18.121730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:18.122020 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:18.122160 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:18.123444 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0619 16:55:18.123647 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":410,"threads_started":1}
I20260619 16:55:18.126520 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:18.094924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40121 (local address 127.16.48.1:35131)
0619 16:55:18.095866 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:18.095931 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:18.098991 (+ 3060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:18.100209 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:18.100232 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:18.101390 (+ 1158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:18.102636 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:18.102667 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.104240 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.104248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:18.107418 (+ 3170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:18.107453 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.109277 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.109289 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:18.109740 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:18.110749 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:18.110807 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:18.115281 (+ 4474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:18.119589 (+ 4308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:18.119606 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:18.119635 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:18.120627 (+ 992us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:18.121134 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:18.121140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:18.121155 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:18.121537 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:18.122192 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:18.122206 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:18.122881 (+ 675us) client_negotiation.cc:769] Sending connection context
0619 16:55:18.123315 (+ 434us) client_negotiation.cc:240] Negotiation successful
0619 16:55:18.124322 (+ 1007us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":357,"threads_started":1}
I20260619 16:55:18.150755 17775 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:40935
I20260619 16:55:18.150775 17910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:40935 every 8 connection(s)
I20260619 16:55:18.152456 17775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:18.155695 17911 heartbeater.cc:347] Connected to a master server at 127.16.48.62:40121
I20260619 16:55:18.156180 17911 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:18.157138 17911 heartbeater.cc:511] Master 127.16.48.62:40121 requested a full tablet report, sending...
I20260619 16:55:18.158047 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 17775
I20260619 16:55:18.158308 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:18.159134 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:18.142998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:46971 (local address 127.16.48.62:40121)
0619 16:55:18.143225 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:18.143232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:18.144175 (+ 943us) server_negotiation.cc:407] Connection header received
0619 16:55:18.144542 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:18.144547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:18.144616 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:18.144753 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:18.146612 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.147730 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.149592 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.150188 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.150345 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:18.152740 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:18.152745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:18.152749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:18.152785 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:18.155666 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:18.156673 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:18.156679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:18.156681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:18.156834 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:18.157428 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:18.157433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:18.157437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:18.157857 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:18.158044 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:18.158654 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 16:55:18.158882 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:55:18.159128 16576 external_mini_cluster.cc:1428] Setting key d36f62ab1dd66607c8c5bf0db0c9529c
I20260619 16:55:18.159446 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:18.143025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40121 (local address 127.16.48.1:46971)
0619 16:55:18.143986 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:18.144015 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:18.144355 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:18.144982 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:18.144993 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:18.145842 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:18.146420 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:18.146435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.147940 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.147944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:18.149386 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:18.149403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.150432 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.150439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:18.150628 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:18.151469 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:18.151490 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:18.152432 (+ 942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:18.155829 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:18.155834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:18.155838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:18.156507 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:18.157066 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:18.157071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:18.157074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:18.157239 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:18.158057 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:18.158062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:18.158339 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 16:55:18.158522 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 16:55:18.158919 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":557,"threads_started":1}
I20260619 16:55:18.161696 17704 ts_manager.cc:194] Registered new tserver with Master: a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1:40935)
I20260619 16:55:18.163776 17704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:35131
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:55:18.203962 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:40121
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:18.550024 17915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:18.550424 17915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:18.550602 17915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:18.559021 17915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:18.559208 17915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:18.559360 17915 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:55:18.559511 17915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:18.570297 17915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.48.62:40121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:18.571892 17915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:18.573402 17915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:18.588429 17923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:18.588455 17921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:18.588505 17920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:18.589110 17915 server_base.cc:1034] running on GCE node
I20260619 16:55:18.590109 17915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:18.591209 17915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:18.592475 17915 hybrid_clock.cc:648] HybridClock initialized: now 1781888118592408 us; error 59 us; skew 500 ppm
I20260619 16:55:18.592820 17915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:18 dist-test-slave-3ffd krb5kdc[17659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888118, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:18.596571 17915 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:55:18.598284 17915 webserver.cc:460] Webserver started at http://127.16.48.2:42049/ using document root <none> and password file <none>
I20260619 16:55:18.599026 17915 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:18.599119 17915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:18.599411 17915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:18.602608 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "3cb756c0a4b84f7291e0292af91db279"
format_stamp: "Formatted at 2026-06-19 16:55:18 on dist-test-slave-3ffd"
server_key: "5d49f43a4ae6d500e402645855577d9e"
server_key_iv: "90484bb11219f7751379d474bdfa4d1d"
server_key_version: "encryptionkey@0"
I20260619 16:55:18.603451 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "3cb756c0a4b84f7291e0292af91db279"
format_stamp: "Formatted at 2026-06-19 16:55:18 on dist-test-slave-3ffd"
server_key: "5d49f43a4ae6d500e402645855577d9e"
server_key_iv: "90484bb11219f7751379d474bdfa4d1d"
server_key_version: "encryptionkey@0"
I20260619 16:55:18.609422 17915 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260619 16:55:18.613528 17930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:18.615222 17915 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260619 16:55:18.615445 17915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "3cb756c0a4b84f7291e0292af91db279"
format_stamp: "Formatted at 2026-06-19 16:55:18 on dist-test-slave-3ffd"
server_key: "5d49f43a4ae6d500e402645855577d9e"
server_key_iv: "90484bb11219f7751379d474bdfa4d1d"
server_key_version: "encryptionkey@0"
I20260619 16:55:18.615729 17915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:18.638864 17915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:18.643586 17915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:18.643895 17915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:18.646862 17915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:18.646953 17915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:18.647159 17915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:18.647249 17915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:55:18 dist-test-slave-3ffd krb5kdc[17659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888118, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:18.676359 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:18.647512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:40701 (local address 127.16.48.62:40121)
0619 16:55:18.647718 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:18.647723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:18.649651 (+ 1928us) server_negotiation.cc:407] Connection header received
0619 16:55:18.652909 (+ 3258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:18.652916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:18.653017 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:18.653146 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:18.656112 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.657099 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.660532 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.661159 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.661328 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:18.668042 (+ 6714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:18.668050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:18.668056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:18.668164 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:18.671746 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:18.672957 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:18.672962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:18.672964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:18.673100 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:18.673874 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:18.673883 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:18.673889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:18.674408 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:18.674630 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:18.675458 (+ 828us) server_negotiation.cc:299] Negotiation successful
0619 16:55:18.675795 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:55:18.677551 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:18.648417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40121 (local address 127.16.48.2:40701)
0619 16:55:18.649435 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:18.649494 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:18.652410 (+ 2916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:18.653500 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:18.653520 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:18.654548 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:18.655814 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:18.655853 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.657313 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.657322 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:18.660308 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:18.660335 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.661507 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.661518 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:18.661956 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:18.662969 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:18.663018 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:18.667635 (+ 4617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:18.671942 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:18.671954 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:18.671979 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:18.672745 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:18.673247 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:18.673255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:18.673259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:18.673622 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:18.674598 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:18.674606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:18.675033 (+ 427us) client_negotiation.cc:769] Sending connection context
0619 16:55:18.675297 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 16:55:18.675732 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":412,"threads_started":1}
I20260619 16:55:18.703750 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:18.685515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40121 (local address 127.16.48.2:58743)
0619 16:55:18.686195 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:18.686217 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:18.686357 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:18.686788 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:18.686791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:18.687224 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:18.687515 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:18.687524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.689294 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.689302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:18.690771 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:18.690791 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.692051 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.692061 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:18.692274 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:18.693053 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:18.693071 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:18.693872 (+ 801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:18.700013 (+ 6141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:18.700021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:18.700027 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:18.700852 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:18.701549 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:18.701559 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:18.701566 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:18.701796 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:18.702743 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:18.702756 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:18.702957 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 16:55:18.703094 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 16:55:18.703394 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":386,"threads_started":1}
I20260619 16:55:18.704196 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:18.685495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:58743 (local address 127.16.48.62:40121)
0619 16:55:18.685758 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:18.685768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:18.686322 (+ 554us) server_negotiation.cc:407] Connection header received
0619 16:55:18.686489 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:18.686497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:18.686602 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:18.686761 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:18.687673 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.689095 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.690980 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:18.691789 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:18.691954 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:18.694129 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:18.694139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:18.694147 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:18.694196 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:18.699765 (+ 5569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:18.701118 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:18.701126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:18.701131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:18.701362 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:18.701992 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:18.701997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:18.702000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:18.702560 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:18.702749 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:18.703258 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 16:55:18.703588 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:55:18.711331 17915 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:41969
I20260619 16:55:18.711337 18049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:41969 every 8 connection(s)
I20260619 16:55:18.712770 17915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:18.714764 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 17915
I20260619 16:55:18.714977 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:18.715031 18050 heartbeater.cc:347] Connected to a master server at 127.16.48.62:40121
I20260619 16:55:18.715278 18050 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:18.715550 16576 external_mini_cluster.cc:1428] Setting key 7763de1060ccff2ace284e727f7d57b4
I20260619 16:55:18.716180 18050 heartbeater.cc:511] Master 127.16.48.62:40121 requested a full tablet report, sending...
I20260619 16:55:18.717672 17704 ts_manager.cc:194] Registered new tserver with Master: 3cb756c0a4b84f7291e0292af91db279 (127.16.48.2:41969)
I20260619 16:55:18.718561 17704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:40701
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:55:18.759352 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:40121
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:19.103928 18054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:19.104269 18054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:19.104368 18054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:19.112021 18054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:19.112221 18054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:19.112325 18054 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:55:19.112412 18054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:19.123435 18054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.48.62:40121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:19.125072 18054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:19.126649 18054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:19.141357 18059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:19.141428 18060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:19.141402 18062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:19.142509 18054 server_base.cc:1034] running on GCE node
I20260619 16:55:19.143234 18054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:19.144385 18054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:19.145699 18054 hybrid_clock.cc:648] HybridClock initialized: now 1781888119145598 us; error 71 us; skew 500 ppm
I20260619 16:55:19.146063 18054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:19 dist-test-slave-3ffd krb5kdc[17659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:19.149662 18054 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:55:19.151472 18054 webserver.cc:460] Webserver started at http://127.16.48.3:40175/ using document root <none> and password file <none>
I20260619 16:55:19.152329 18054 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:19.152474 18054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:19.152828 18054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:19.156345 18054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0"
format_stamp: "Formatted at 2026-06-19 16:55:19 on dist-test-slave-3ffd"
server_key: "ec3317385fb31e28e61dc8b45109183c"
server_key_iv: "5e54598bc43b2ab4c0baaa34f32d02a7"
server_key_version: "encryptionkey@0"
I20260619 16:55:19.157315 18054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0"
format_stamp: "Formatted at 2026-06-19 16:55:19 on dist-test-slave-3ffd"
server_key: "ec3317385fb31e28e61dc8b45109183c"
server_key_iv: "5e54598bc43b2ab4c0baaa34f32d02a7"
server_key_version: "encryptionkey@0"
I20260619 16:55:19.163379 18054 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 16:55:19.167361 17911 heartbeater.cc:503] Master 127.16.48.62:40121 was elected leader, sending a full tablet report...
I20260619 16:55:19.167963 18069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:19.169876 18054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:55:19.170110 18054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0"
format_stamp: "Formatted at 2026-06-19 16:55:19 on dist-test-slave-3ffd"
server_key: "ec3317385fb31e28e61dc8b45109183c"
server_key_iv: "5e54598bc43b2ab4c0baaa34f32d02a7"
server_key_version: "encryptionkey@0"
I20260619 16:55:19.170373 18054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:19.185866 18054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:19.196573 18054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:19.196833 18054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:19.199903 18054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:19.200026 18054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:19.200240 18054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:19.200327 18054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:55:19 dist-test-slave-3ffd krb5kdc[17659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:19.228173 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.200599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:48729 (local address 127.16.48.62:40121)
0619 16:55:19.200854 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.200862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.202591 (+ 1729us) server_negotiation.cc:407] Connection header received
0619 16:55:19.205164 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.205173 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.205315 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.205473 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:19.207981 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.209139 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.212032 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.212561 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.212715 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.218437 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:19.218448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:19.218455 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:19.218510 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:19.222981 (+ 4471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:19.224385 (+ 1404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:19.224392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:19.224396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:19.224579 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:19.225344 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:19.225352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:19.225357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:19.225803 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:19.225983 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.227361 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.227669 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:55:19.230325 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.201407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40121 (local address 127.16.48.3:48729)
0619 16:55:19.202388 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:19.202434 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:19.204726 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:19.205833 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:19.205846 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:19.206704 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:19.207766 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.207790 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.209316 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.209322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.211807 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.211826 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.212845 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.212856 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.213206 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.214124 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:19.214167 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:19.217995 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:19.223228 (+ 5233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:19.223242 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:19.223267 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:19.224194 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:19.224806 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:19.224817 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:19.224821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:19.225164 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:19.226007 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:19.226017 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:19.226711 (+ 694us) client_negotiation.cc:769] Sending connection context
0619 16:55:19.227199 (+ 488us) client_negotiation.cc:240] Negotiation successful
0619 16:55:19.228020 (+ 821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":432,"threads_started":1}
I20260619 16:55:19.243927 18054 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:41557
I20260619 16:55:19.243988 18188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:41557 every 8 connection(s)
I20260619 16:55:19.245576 18054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:19.248384 18189 heartbeater.cc:347] Connected to a master server at 127.16.48.62:40121
I20260619 16:55:19.248670 18189 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:19.249491 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 18054
I20260619 16:55:19.249562 18189 heartbeater.cc:511] Master 127.16.48.62:40121 requested a full tablet report, sending...
I20260619 16:55:19.249692 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:19.250165 16576 external_mini_cluster.cc:1428] Setting key c6193d1275993402cc37e29e7b233216
I20260619 16:55:19.251470 17704 ts_manager.cc:194] Registered new tserver with Master: 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557)
I20260619 16:55:19.252799 17704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:48729
I20260619 16:55:19.253510 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:55:19.255455 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.239172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40121 (local address 127.16.48.3:49473)
0619 16:55:19.240361 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:19.240388 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:19.240547 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:19.240966 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:19.240974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:19.241621 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:19.242142 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.242154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.243799 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.243805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.245044 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.245059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.246094 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.246141 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.246398 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.247272 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:19.247294 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:19.248208 (+ 914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:19.252460 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:19.252466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:19.252470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:19.253011 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:19.253488 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:19.253496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:19.253500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:19.253653 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:19.254352 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:19.254356 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:19.254540 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 16:55:19.254681 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 16:55:19.254997 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":773,"threads_started":1}
I20260619 16:55:19.256383 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.239181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:49473 (local address 127.16.48.62:40121)
0619 16:55:19.239427 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.239435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.240533 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 16:55:19.240669 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.240677 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.240777 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.240911 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:19.242323 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.243570 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.245223 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.245875 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.246019 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.248422 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:19.248427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:19.248431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:19.248486 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:19.252273 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:19.253178 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:19.253185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:19.253189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:19.253347 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:19.253782 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:19.253787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:19.253790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:19.254111 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:19.254290 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.255791 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.256037 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 16:55:19 dist-test-slave-3ffd krb5kdc[17659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:55:19 dist-test-slave-3ffd krb5kdc[17659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:19.304294 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.288666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.16.48.62:40121)
0619 16:55:19.288912 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.288919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.289217 (+ 298us) server_negotiation.cc:407] Connection header received
0619 16:55:19.289439 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.289444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.289526 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.289641 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:19.291244 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.292077 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.293457 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.293928 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.294082 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.298142 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:19.298147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:19.298151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:19.298190 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:19.300970 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:19.301843 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:19.301852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:19.301861 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:19.302029 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:19.302524 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:19.302533 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:19.302541 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:19.302861 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:19.303142 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.303519 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.303785 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:55:19.308957 17704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:55:19.316352 17704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:19.355736 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.331560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.16.48.2:41969)
0619 16:55:19.345265 (+ 13705us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.345272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.346123 (+ 851us) server_negotiation.cc:407] Connection header received
0619 16:55:19.346234 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.346241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.346521 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.346741 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:19.347851 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.349494 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.352919 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.354433 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.354582 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.354795 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.355222 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.355414 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13524,"thread_start_us":13379,"threads_started":1}
I20260619 16:55:19.356627 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.330948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41969 (local address 127.0.0.1:47750)
0619 16:55:19.345798 (+ 14850us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:19.345819 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:19.345963 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:19.346919 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:19.346923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:19.346944 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:19.347586 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.347619 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.349713 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.349719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.352722 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.352740 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.354670 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.354680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.354927 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.354975 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:55:19.355159 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 16:55:19.356192 (+ 1033us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14704,"mutex_wait_us":1093,"thread_start_us":9766,"threads_started":1}
I20260619 16:55:19.357213 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.330823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.16.48.3:41557)
0619 16:55:19.344973 (+ 14150us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.344987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.345042 (+ 55us) server_negotiation.cc:407] Connection header received
0619 16:55:19.345566 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.345576 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.345911 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.346170 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:19.347883 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.349522 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.353472 (+ 3950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.355031 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.355174 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.355421 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.356677 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.356879 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13888,"thread_start_us":13667,"threads_started":1}
I20260619 16:55:19.357434 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.330599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:41557 (local address 127.0.0.1:59418)
0619 16:55:19.341161 (+ 10562us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:19.341211 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:19.341437 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:19.346713 (+ 5276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:19.346724 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:19.346773 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:19.347645 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.347658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.349713 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.349719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.353276 (+ 3557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.353294 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.355571 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.355578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.355817 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.355850 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:55:19.355942 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:55:19.356094 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10282,"mutex_wait_us":265,"thread_start_us":14864,"threads_started":1}
I20260619 16:55:19.360458 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.347298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:40935 (local address 127.0.0.1:60160)
0619 16:55:19.352181 (+ 4883us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:19.352201 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:19.352384 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:19.353056 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:19.353061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:19.353088 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:19.353519 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.353527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.354903 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.354909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.358148 (+ 3239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.358165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.359621 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.359627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.359797 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.359837 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:55:19.359959 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 16:55:19.360095 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4693,"thread_start_us":4513,"threads_started":1}
I20260619 16:55:19.360515 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.346235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.16.48.1:40935)
0619 16:55:19.351833 (+ 5598us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.351843 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.352332 (+ 489us) server_negotiation.cc:407] Connection header received
0619 16:55:19.352566 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.352573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.352803 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.353067 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:19.353691 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.354731 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.358362 (+ 3631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.359403 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.359567 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.359785 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.360012 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.360234 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5415,"thread_start_us":4276,"threads_started":1}
I20260619 16:55:19.361017 17980 tablet_service.cc:1467] Processing CreateTablet for tablet 353a929d31944bd78799f1bb0437fc72 (DEFAULT_TABLE table=test-table [id=4d8a6114a77f464584c3cd362af4b0c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:19.361017 18119 tablet_service.cc:1467] Processing CreateTablet for tablet 353a929d31944bd78799f1bb0437fc72 (DEFAULT_TABLE table=test-table [id=4d8a6114a77f464584c3cd362af4b0c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:19.362622 18119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353a929d31944bd78799f1bb0437fc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:19.362624 17980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353a929d31944bd78799f1bb0437fc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:19.367483 17841 tablet_service.cc:1467] Processing CreateTablet for tablet 353a929d31944bd78799f1bb0437fc72 (DEFAULT_TABLE table=test-table [id=4d8a6114a77f464584c3cd362af4b0c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:19.369040 17841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353a929d31944bd78799f1bb0437fc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:19.379506 18208 tablet_bootstrap.cc:492] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: Bootstrap starting.
I20260619 16:55:19.383623 18208 tablet_bootstrap.cc:654] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:19.384501 18210 tablet_bootstrap.cc:492] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: Bootstrap starting.
I20260619 16:55:19.385260 18208 log.cc:824] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:19.386114 18209 tablet_bootstrap.cc:492] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Bootstrap starting.
I20260619 16:55:19.388345 18208 tablet_bootstrap.cc:492] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: No bootstrap required, opened a new log
I20260619 16:55:19.388666 18208 ts_tablet_manager.cc:1397] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 16:55:19.389003 18210 tablet_bootstrap.cc:654] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:19.389919 18209 tablet_bootstrap.cc:654] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:19.390759 18210 log.cc:824] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:19.391609 18209 log.cc:824] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:19.395150 18210 tablet_bootstrap.cc:492] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: No bootstrap required, opened a new log
I20260619 16:55:19.395633 18210 ts_tablet_manager.cc:1397] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260619 16:55:19.397408 18208 raft_consensus.cc:348] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } }
I20260619 16:55:19.397825 18208 raft_consensus.cc:374] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:19.397940 18208 raft_consensus.cc:729] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb756c0a4b84f7291e0292af91db279, State: Initialized, Role: FOLLOWER
I20260619 16:55:19.398612 18208 consensus_queue.cc:260] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } }
I20260619 16:55:19.398809 18209 tablet_bootstrap.cc:492] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: No bootstrap required, opened a new log
I20260619 16:55:19.399151 18209 ts_tablet_manager.cc:1397] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260619 16:55:19.399682 18050 heartbeater.cc:503] Master 127.16.48.62:40121 was elected leader, sending a full tablet report...
I20260619 16:55:19.401782 18208 ts_tablet_manager.cc:1428] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260619 16:55:19.407644 18210 raft_consensus.cc:348] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } }
I20260619 16:55:19.408179 18210 raft_consensus.cc:374] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:19.408303 18210 raft_consensus.cc:729] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2337128874c44b6ac7610cd44bf8f78, State: Initialized, Role: FOLLOWER
I20260619 16:55:19.409024 18210 consensus_queue.cc:260] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } }
I20260619 16:55:19.410449 18209 raft_consensus.cc:348] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } }
I20260619 16:55:19.410936 18210 ts_tablet_manager.cc:1428] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260619 16:55:19.411000 18209 raft_consensus.cc:374] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:19.411142 18209 raft_consensus.cc:729] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8a68e33b5a4b2393c8f2b7f278c4e0, State: Initialized, Role: FOLLOWER
I20260619 16:55:19.411782 18209 consensus_queue.cc:260] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } }
I20260619 16:55:19.412972 18189 heartbeater.cc:503] Master 127.16.48.62:40121 was elected leader, sending a full tablet report...
I20260619 16:55:19.413630 18209 ts_tablet_manager.cc:1428] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260619 16:55:19.460222 18214 raft_consensus.cc:482] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:19.460541 18214 raft_consensus.cc:504] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } }
I20260619 16:55:19.462219 18214 leader_election.cc:288] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557), a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1:40935)
W20260619 16:55:19.464929 18051 tablet.cc:2369] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:55:19.470697 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.462789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:59683 (local address 127.16.48.3:41557)
0619 16:55:19.463038 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.463047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.463586 (+ 539us) server_negotiation.cc:407] Connection header received
0619 16:55:19.463860 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.463865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.463949 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.464023 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:19.464892 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.466461 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.468712 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.469639 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.469784 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.469910 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.470287 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.470432 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:55:19.470748 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.462663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:41557 (local address 127.16.48.2:59683)
0619 16:55:19.463453 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:19.463475 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:19.463703 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:19.464275 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:19.464279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:19.464307 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:19.464725 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.464736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.466665 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.466671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.468529 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.468546 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.469901 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.469907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.470094 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.470130 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:55:19.470226 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:55:19.470372 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":445,"threads_started":1}
I20260619 16:55:19.471768 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.464281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:47175 (local address 127.16.48.1:40935)
0619 16:55:19.464499 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.464504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.464751 (+ 247us) server_negotiation.cc:407] Connection header received
0619 16:55:19.465027 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.465034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.465128 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.465194 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:19.466081 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.466957 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.469526 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.470624 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.470784 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.470889 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.471465 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.471581 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:55:19.471988 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353a929d31944bd78799f1bb0437fc72" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" is_pre_election: true
I20260619 16:55:19.472496 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.463587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:40935 (local address 127.16.48.2:47175)
0619 16:55:19.464633 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:19.464654 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:19.464851 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:19.465427 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:19.465434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:19.465460 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:19.465900 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.465913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.467110 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.467115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.469346 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.469361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.471077 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.471083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.471271 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.471303 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:55:19.471973 (+ 670us) client_negotiation.cc:240] Negotiation successful
0619 16:55:19.472190 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":695,"threads_started":1}
I20260619 16:55:19.472610 18139 raft_consensus.cc:2454] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 0.
I20260619 16:55:19.473359 17934 leader_election.cc:302] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8a68e33b5a4b2393c8f2b7f278c4e0, 3cb756c0a4b84f7291e0292af91db279; no voters:
I20260619 16:55:19.473570 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353a929d31944bd78799f1bb0437fc72" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2337128874c44b6ac7610cd44bf8f78" is_pre_election: true
I20260619 16:55:19.473778 18214 raft_consensus.cc:2781] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:19.473956 18214 raft_consensus.cc:482] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:19.474086 18214 raft_consensus.cc:3037] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:19.474150 17861 raft_consensus.cc:2454] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 0.
I20260619 16:55:19.476693 18214 raft_consensus.cc:504] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } }
I20260619 16:55:19.477807 18214 leader_election.cc:288] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 election: Requested vote from peers 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557), a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1:40935)
I20260619 16:55:19.478426 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353a929d31944bd78799f1bb0437fc72" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0"
I20260619 16:55:19.478451 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353a929d31944bd78799f1bb0437fc72" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2337128874c44b6ac7610cd44bf8f78"
I20260619 16:55:19.478683 18139 raft_consensus.cc:3037] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:19.478729 17861 raft_consensus.cc:3037] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:19.481508 18139 raft_consensus.cc:2454] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 1.
I20260619 16:55:19.481966 17861 raft_consensus.cc:2454] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 1.
I20260619 16:55:19.481983 17934 leader_election.cc:302] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8a68e33b5a4b2393c8f2b7f278c4e0, 3cb756c0a4b84f7291e0292af91db279; no voters:
I20260619 16:55:19.482313 18214 raft_consensus.cc:2781] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:19.482821 18214 raft_consensus.cc:686] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb756c0a4b84f7291e0292af91db279, State: Running, Role: LEADER
I20260619 16:55:19.483382 18214 consensus_queue.cc:237] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } }
I20260619 16:55:19.489077 17702 catalog_manager.cc:5495] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cb756c0a4b84f7291e0292af91db279 (127.16.48.2). New cstate: current_term: 1 leader_uuid: "3cb756c0a4b84f7291e0292af91db279" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } health_report { overall_health: UNKNOWN } } }
W20260619 16:55:19.497594 18190 tablet.cc:2369] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:55:19.514956 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.508963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.16.48.2:41969)
0619 16:55:19.509185 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.509192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.509352 (+ 160us) server_negotiation.cc:407] Connection header received
0619 16:55:19.509705 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.509709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.509799 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.509955 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:55:19.510321 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.511049 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.512498 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.512961 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.513117 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.513674 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:55:19.514268 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:55:19.514441 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.514579 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.514669 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:55:19.524425 18139 raft_consensus.cc:1261] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 1 FOLLOWER]: Refusing update from remote peer 3cb756c0a4b84f7291e0292af91db279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:19.524425 17861 raft_consensus.cc:1261] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Refusing update from remote peer 3cb756c0a4b84f7291e0292af91db279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:19.525619 18219 consensus_queue.cc:1035] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:19.526086 18214 consensus_queue.cc:1035] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:19.568518 17704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:19.568872 17704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:19.573482 17704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2: Sending DeleteTablet for 3 replicas of tablet 353a929d31944bd78799f1bb0437fc72
I20260619 16:55:19.574926 17841 tablet_service.cc:1514] Processing DeleteTablet for tablet 353a929d31944bd78799f1bb0437fc72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:19 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:60160
I20260619 16:55:19.575124 17980 tablet_service.cc:1514] Processing DeleteTablet for tablet 353a929d31944bd78799f1bb0437fc72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:19 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:47750
I20260619 16:55:19.576418 18119 tablet_service.cc:1514] Processing DeleteTablet for tablet 353a929d31944bd78799f1bb0437fc72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:19 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:59418
I20260619 16:55:19.577569 18234 tablet_replica.cc:330] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: stopping tablet replica
I20260619 16:55:19.577690 17702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:55:19.578159 18234 raft_consensus.cc:2229] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:19.578742 18234 raft_consensus.cc:2258] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:55:19.579279 17702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:19.581826 18234 ts_tablet_manager.cc:1905] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:19.587863 18232 tablet_replica.cc:330] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: stopping tablet replica
I20260619 16:55:19.588443 18232 raft_consensus.cc:2229] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:19.588989 18232 raft_consensus.cc:2258] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:19.590952 17979 tablet_service.cc:1467] Processing CreateTablet for tablet 07e87978df9c4e528a9dd3a859d78fe2 (DEFAULT_TABLE table=test-table [id=e0026d2f12e842fd8bf26bbcbbc90f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:19.591593 17979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e87978df9c4e528a9dd3a859d78fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:19.591823 17840 tablet_service.cc:1467] Processing CreateTablet for tablet 07e87978df9c4e528a9dd3a859d78fe2 (DEFAULT_TABLE table=test-table [id=e0026d2f12e842fd8bf26bbcbbc90f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:19.590821 18234 ts_tablet_manager.cc:1918] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:55:19.592290 17840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e87978df9c4e528a9dd3a859d78fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:19.592411 18234 log.cc:1196] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/353a929d31944bd78799f1bb0437fc72
I20260619 16:55:19.593075 18234 ts_tablet_manager.cc:1939] T 353a929d31944bd78799f1bb0437fc72 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Deleting consensus metadata
I20260619 16:55:19.596889 18233 tablet_replica.cc:330] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: stopping tablet replica
I20260619 16:55:19.596871 17691 catalog_manager.cc:4841] TS 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557): tablet 353a929d31944bd78799f1bb0437fc72 (table test-table [id=4d8a6114a77f464584c3cd362af4b0c5]) successfully deleted
I20260619 16:55:19.597777 18119 tablet_service.cc:1467] Processing CreateTablet for tablet 07e87978df9c4e528a9dd3a859d78fe2 (DEFAULT_TABLE table=test-table [id=e0026d2f12e842fd8bf26bbcbbc90f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:19.598244 18119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e87978df9c4e528a9dd3a859d78fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:19.599532 18232 ts_tablet_manager.cc:1905] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:19.602831 18210 tablet_bootstrap.cc:492] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: Bootstrap starting.
I20260619 16:55:19.603958 18208 tablet_bootstrap.cc:492] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: Bootstrap starting.
I20260619 16:55:19.605351 18210 tablet_bootstrap.cc:654] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:19.605722 18208 tablet_bootstrap.cc:654] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:19.605945 18209 tablet_bootstrap.cc:492] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Bootstrap starting.
I20260619 16:55:19.608208 18232 ts_tablet_manager.cc:1918] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:55:19.608431 18209 tablet_bootstrap.cc:654] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:19.608467 18232 log.cc:1196] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/353a929d31944bd78799f1bb0437fc72
I20260619 16:55:19.610062 18233 raft_consensus.cc:2229] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:55:19.611217 18232 ts_tablet_manager.cc:1939] T 353a929d31944bd78799f1bb0437fc72 P a2337128874c44b6ac7610cd44bf8f78: Deleting consensus metadata
I20260619 16:55:19.612427 18233 raft_consensus.cc:2258] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:19.615662 17691 catalog_manager.cc:4841] TS a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1:40935): tablet 353a929d31944bd78799f1bb0437fc72 (table test-table [id=4d8a6114a77f464584c3cd362af4b0c5]) successfully deleted
I20260619 16:55:19.617529 18208 tablet_bootstrap.cc:492] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: No bootstrap required, opened a new log
I20260619 16:55:19.617736 18208 ts_tablet_manager.cc:1397] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260619 16:55:19.618721 18208 raft_consensus.cc:348] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.618999 18208 raft_consensus.cc:374] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:19.619119 18208 raft_consensus.cc:729] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb756c0a4b84f7291e0292af91db279, State: Initialized, Role: FOLLOWER
I20260619 16:55:19.619436 18208 consensus_queue.cc:260] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.619997 18209 tablet_bootstrap.cc:492] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: No bootstrap required, opened a new log
I20260619 16:55:19.620194 18209 ts_tablet_manager.cc:1397] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260619 16:55:19.620299 18233 ts_tablet_manager.cc:1905] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:19.621131 18209 raft_consensus.cc:348] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.621367 18209 raft_consensus.cc:374] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:19.621469 18209 raft_consensus.cc:729] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8a68e33b5a4b2393c8f2b7f278c4e0, State: Initialized, Role: FOLLOWER
I20260619 16:55:19.621775 18209 consensus_queue.cc:260] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.622545 18208 ts_tablet_manager.cc:1428] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:55:19.624256 18210 tablet_bootstrap.cc:492] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: No bootstrap required, opened a new log
I20260619 16:55:19.624446 18210 ts_tablet_manager.cc:1397] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.008s
I20260619 16:55:19.624750 18209 ts_tablet_manager.cc:1428] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:55:19.625921 18210 raft_consensus.cc:348] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.626184 18210 raft_consensus.cc:374] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:19.626296 18210 raft_consensus.cc:729] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2337128874c44b6ac7610cd44bf8f78, State: Initialized, Role: FOLLOWER
I20260619 16:55:19.626600 18210 consensus_queue.cc:260] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.627257 18210 ts_tablet_manager.cc:1428] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:55:19.633550 18233 ts_tablet_manager.cc:1918] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:55:19.633782 18233 log.cc:1196] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/353a929d31944bd78799f1bb0437fc72
I20260619 16:55:19.634395 18233 ts_tablet_manager.cc:1939] T 353a929d31944bd78799f1bb0437fc72 P 3cb756c0a4b84f7291e0292af91db279: Deleting consensus metadata
I20260619 16:55:19.636827 17688 catalog_manager.cc:4841] TS 3cb756c0a4b84f7291e0292af91db279 (127.16.48.2:41969): tablet 353a929d31944bd78799f1bb0437fc72 (table test-table [id=4d8a6114a77f464584c3cd362af4b0c5]) successfully deleted
W20260619 16:55:19.657721 17912 tablet.cc:2369] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:55:19.746510 18219 raft_consensus.cc:482] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:19.746814 18219 raft_consensus.cc:504] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.747468 18219 leader_election.cc:288] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1:40935), 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557)
I20260619 16:55:19.748250 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e87978df9c4e528a9dd3a859d78fe2" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" is_pre_election: true
I20260619 16:55:19.748241 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e87978df9c4e528a9dd3a859d78fe2" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2337128874c44b6ac7610cd44bf8f78" is_pre_election: true
I20260619 16:55:19.748498 18139 raft_consensus.cc:2454] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 0.
I20260619 16:55:19.748518 17861 raft_consensus.cc:2454] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 0.
I20260619 16:55:19.749032 17934 leader_election.cc:302] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8a68e33b5a4b2393c8f2b7f278c4e0, 3cb756c0a4b84f7291e0292af91db279; no voters:
I20260619 16:55:19.749415 18219 raft_consensus.cc:2781] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:19.749516 18219 raft_consensus.cc:482] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:19.749722 18219 raft_consensus.cc:3037] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:19.752251 18219 raft_consensus.cc:504] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.752946 18219 leader_election.cc:288] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 election: Requested vote from peers a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1:40935), 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557)
I20260619 16:55:19.753687 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e87978df9c4e528a9dd3a859d78fe2" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2337128874c44b6ac7610cd44bf8f78"
I20260619 16:55:19.753691 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e87978df9c4e528a9dd3a859d78fe2" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0"
I20260619 16:55:19.753942 18139 raft_consensus.cc:3037] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:19.753947 17861 raft_consensus.cc:3037] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:19.756559 17861 raft_consensus.cc:2454] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 1.
I20260619 16:55:19.757014 18139 raft_consensus.cc:2454] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 1.
I20260619 16:55:19.757256 17934 leader_election.cc:302] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb756c0a4b84f7291e0292af91db279, a2337128874c44b6ac7610cd44bf8f78; no voters:
I20260619 16:55:19.757685 18219 raft_consensus.cc:2781] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:19.757908 18219 raft_consensus.cc:686] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb756c0a4b84f7291e0292af91db279, State: Running, Role: LEADER
I20260619 16:55:19.758219 18219 consensus_queue.cc:237] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.761858 17704 catalog_manager.cc:5495] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cb756c0a4b84f7291e0292af91db279 (127.16.48.2). New cstate: current_term: 1 leader_uuid: "3cb756c0a4b84f7291e0292af91db279" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:55:19 dist-test-slave-3ffd krb5kdc[17659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888117, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:19.814298 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.799258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.16.48.62:40121)
0619 16:55:19.799478 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:19.799486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:19.800102 (+ 616us) server_negotiation.cc:407] Connection header received
0619 16:55:19.800335 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:19.800344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:19.800398 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:19.800504 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:19.801713 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.802430 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.803716 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.804217 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.804355 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.808067 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:19.808072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:19.808075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:19.808106 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:19.811112 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:19.812110 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:19.812160 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:19.812167 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:19.812315 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:19.812749 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:19.812752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:19.812754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:19.813020 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:19.813130 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:19.813692 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 16:55:19.814033 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":73}
I20260619 16:55:19.814386 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:19.799247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40121 (local address 127.0.0.1:57584)
0619 16:55:19.799953 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:19.799982 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:19.800203 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:19.800527 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:19.800531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:19.801275 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:19.801535 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.801545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.802574 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.802581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.803561 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:19.803573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:19.804395 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:19.804401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:19.804559 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:19.805311 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:19.805328 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:19.807835 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:19.811293 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:19.811312 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:19.811321 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:19.811980 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:19.812433 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:19.812438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:19.812441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:19.812637 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:19.813190 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:19.813198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:19.813395 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 16:55:19.813582 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 16:55:19.813909 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":407,"threads_started":1}
I20260619 16:55:19.820654 17704 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:57584:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:55:19.821766 17704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:19.829681 17840 tablet_service.cc:1467] Processing CreateTablet for tablet 4849780340cc458cba119e10238d3565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de7a2f217cd34d649f32ef18eb6de88a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:55:19.830008 17979 tablet_service.cc:1467] Processing CreateTablet for tablet 4849780340cc458cba119e10238d3565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de7a2f217cd34d649f32ef18eb6de88a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:55:19.830322 17840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4849780340cc458cba119e10238d3565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:19.830505 17979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4849780340cc458cba119e10238d3565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:19.833760 18119 tablet_service.cc:1467] Processing CreateTablet for tablet 4849780340cc458cba119e10238d3565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de7a2f217cd34d649f32ef18eb6de88a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:55:19.834372 18119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4849780340cc458cba119e10238d3565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:19.838096 18210 tablet_bootstrap.cc:492] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: Bootstrap starting.
I20260619 16:55:19.838352 18208 tablet_bootstrap.cc:492] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: Bootstrap starting.
I20260619 16:55:19.839891 18210 tablet_bootstrap.cc:654] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:19.839934 18208 tablet_bootstrap.cc:654] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:19.842137 18209 tablet_bootstrap.cc:492] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Bootstrap starting.
I20260619 16:55:19.843870 18209 tablet_bootstrap.cc:654] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:19.850703 18209 tablet_bootstrap.cc:492] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: No bootstrap required, opened a new log
I20260619 16:55:19.850914 18209 ts_tablet_manager.cc:1397] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260619 16:55:19.851351 18208 tablet_bootstrap.cc:492] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: No bootstrap required, opened a new log
I20260619 16:55:19.851539 18208 ts_tablet_manager.cc:1397] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260619 16:55:19.851753 18210 tablet_bootstrap.cc:492] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: No bootstrap required, opened a new log
I20260619 16:55:19.851758 18209 raft_consensus.cc:348] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.851962 18210 ts_tablet_manager.cc:1397] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260619 16:55:19.852027 18209 raft_consensus.cc:374] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:19.852101 18209 raft_consensus.cc:729] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8a68e33b5a4b2393c8f2b7f278c4e0, State: Initialized, Role: FOLLOWER
I20260619 16:55:19.852452 18208 raft_consensus.cc:348] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.852444 18209 consensus_queue.cc:260] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.852706 18208 raft_consensus.cc:374] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:19.852795 18208 raft_consensus.cc:729] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb756c0a4b84f7291e0292af91db279, State: Initialized, Role: FOLLOWER
I20260619 16:55:19.852793 18210 raft_consensus.cc:348] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.853055 18210 raft_consensus.cc:374] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:19.853144 18210 raft_consensus.cc:729] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2337128874c44b6ac7610cd44bf8f78, State: Initialized, Role: FOLLOWER
I20260619 16:55:19.852953 18216 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:19.853104 18208 consensus_queue.cc:260] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.853269 18216 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:19.853392 18210 consensus_queue.cc:260] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:19.853715 18208 ts_tablet_manager.cc:1428] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 16:55:19.853843 18215 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:19.854688 18215 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:19.854635 18219 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:19.854882 18219 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:19.856532 18210 ts_tablet_manager.cc:1428] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260619 16:55:19.858206 18209 ts_tablet_manager.cc:1428] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260619 16:55:20.152383 18215 raft_consensus.cc:482] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:20.152640 18215 raft_consensus.cc:504] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:20.154242 18215 leader_election.cc:288] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb756c0a4b84f7291e0292af91db279 (127.16.48.2:41969), 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557)
I20260619 16:55:20.170814 18219 raft_consensus.cc:482] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:20.171015 18219 raft_consensus.cc:504] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:20.171751 18219 leader_election.cc:288] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1:40935), 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557)
I20260619 16:55:20.172402 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4849780340cc458cba119e10238d3565" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" is_pre_election: true
I20260619 16:55:20.172435 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4849780340cc458cba119e10238d3565" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2337128874c44b6ac7610cd44bf8f78" is_pre_election: true
I20260619 16:55:20.172675 18139 raft_consensus.cc:2454] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 0.
I20260619 16:55:20.172753 17861 raft_consensus.cc:2454] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb756c0a4b84f7291e0292af91db279 in term 0.
I20260619 16:55:20.173362 17934 leader_election.cc:302] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb756c0a4b84f7291e0292af91db279, a2337128874c44b6ac7610cd44bf8f78; no voters:
I20260619 16:55:20.173805 18219 raft_consensus.cc:2781] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:20.173930 18219 raft_consensus.cc:482] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:20.174044 18219 raft_consensus.cc:3037] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:20.174708 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.154781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:60141 (local address 127.16.48.2:41969)
0619 16:55:20.166538 (+ 11757us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.166546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.166573 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:55:20.166660 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.166666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.166749 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.167738 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:20.167900 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.168982 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.171968 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.173273 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.173557 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.173692 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.174112 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.174330 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11605,"thread_start_us":11496,"threads_started":1}
I20260619 16:55:20.174764 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.154453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41969 (local address 127.16.48.1:60141)
0619 16:55:20.164764 (+ 10311us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:20.164801 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:20.165021 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:20.167182 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:20.167187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:20.167230 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:20.167559 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.167571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.169138 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.169145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.171740 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.171760 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.173639 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.173645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.173854 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.173895 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:55:20.174016 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:55:20.174258 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10096,"mutex_wait_us":192,"thread_start_us":12369,"threads_started":1}
I20260619 16:55:20.175690 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.155112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:34297 (local address 127.16.48.3:41557)
0619 16:55:20.167365 (+ 12253us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.167371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.167393 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:55:20.167471 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.167479 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.167561 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.167675 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:20.168628 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.169569 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.171849 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.173160 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.173294 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.173399 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.175255 (+ 1856us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.175417 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12117,"thread_start_us":12025,"threads_started":1}
I20260619 16:55:20.175788 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.154682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:41557 (local address 127.16.48.1:34297)
0619 16:55:20.166776 (+ 12094us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:20.166801 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:20.166910 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:20.167778 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:20.167784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:20.167818 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:20.168390 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.168404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.169712 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.169718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.171612 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.171631 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.173351 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.173358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.173645 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.175070 (+ 1425us) client_negotiation.cc:769] Sending connection context
0619 16:55:20.175193 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:55:20.175359 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11956,"mutex_wait_us":1391,"thread_start_us":9695,"threads_started":1}
I20260619 16:55:20.176460 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4849780340cc458cba119e10238d3565" candidate_uuid: "a2337128874c44b6ac7610cd44bf8f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" is_pre_election: true
I20260619 16:55:20.176731 18139 raft_consensus.cc:2454] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2337128874c44b6ac7610cd44bf8f78 in term 0.
I20260619 16:55:20.177241 17794 leader_election.cc:302] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8a68e33b5a4b2393c8f2b7f278c4e0, a2337128874c44b6ac7610cd44bf8f78; no voters:
I20260619 16:55:20.177663 18215 raft_consensus.cc:2781] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:20.177520 18000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4849780340cc458cba119e10238d3565" candidate_uuid: "a2337128874c44b6ac7610cd44bf8f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb756c0a4b84f7291e0292af91db279" is_pre_election: true
I20260619 16:55:20.177794 18215 raft_consensus.cc:482] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:20.177883 18215 raft_consensus.cc:3037] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:20.180888 18215 raft_consensus.cc:504] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:20.181064 18219 raft_consensus.cc:504] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:20.181609 18000 raft_consensus.cc:2379] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2337128874c44b6ac7610cd44bf8f78 in current term 1: Already voted for candidate 3cb756c0a4b84f7291e0292af91db279 in this term.
I20260619 16:55:20.181794 18215 leader_election.cc:288] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [CANDIDATE]: Term 1 election: Requested vote from peers 3cb756c0a4b84f7291e0292af91db279 (127.16.48.2:41969), 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557)
I20260619 16:55:20.182350 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4849780340cc458cba119e10238d3565" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2337128874c44b6ac7610cd44bf8f78"
I20260619 16:55:20.182694 17861 raft_consensus.cc:2379] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3cb756c0a4b84f7291e0292af91db279 in current term 1: Already voted for candidate a2337128874c44b6ac7610cd44bf8f78 in this term.
I20260619 16:55:20.183156 17998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4849780340cc458cba119e10238d3565" candidate_uuid: "a2337128874c44b6ac7610cd44bf8f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb756c0a4b84f7291e0292af91db279"
I20260619 16:55:20.183369 18219 leader_election.cc:288] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 election: Requested vote from peers a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1:40935), 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557)
I20260619 16:55:20.183534 17998 raft_consensus.cc:2379] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2337128874c44b6ac7610cd44bf8f78 in current term 1: Already voted for candidate 3cb756c0a4b84f7291e0292af91db279 in this term.
I20260619 16:55:20.183655 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4849780340cc458cba119e10238d3565" candidate_uuid: "a2337128874c44b6ac7610cd44bf8f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0"
I20260619 16:55:20.183955 18139 raft_consensus.cc:3037] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:20.184011 18138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4849780340cc458cba119e10238d3565" candidate_uuid: "3cb756c0a4b84f7291e0292af91db279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0"
I20260619 16:55:20.184734 17934 leader_election.cc:302] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb756c0a4b84f7291e0292af91db279; no voters: 1e8a68e33b5a4b2393c8f2b7f278c4e0, a2337128874c44b6ac7610cd44bf8f78
I20260619 16:55:20.185091 18219 raft_consensus.cc:2726] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 16:55:20.186726 18139 raft_consensus.cc:2454] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2337128874c44b6ac7610cd44bf8f78 in term 1.
I20260619 16:55:20.187224 17794 leader_election.cc:302] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e8a68e33b5a4b2393c8f2b7f278c4e0, a2337128874c44b6ac7610cd44bf8f78; no voters: 3cb756c0a4b84f7291e0292af91db279
I20260619 16:55:20.187640 18215 raft_consensus.cc:2781] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:20.188084 18215 raft_consensus.cc:686] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [term 1 LEADER]: Becoming Leader. State: Replica: a2337128874c44b6ac7610cd44bf8f78, State: Running, Role: LEADER
I20260619 16:55:20.188575 18215 consensus_queue.cc:237] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } }
I20260619 16:55:20.190480 18252 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: TxnStatusTablet state changed. Reason: New leader a2337128874c44b6ac7610cd44bf8f78. Latest consensus state: current_term: 1 leader_uuid: "a2337128874c44b6ac7610cd44bf8f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:20.190938 18252 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:55:20.191732 18254 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:55:20.191968 18254 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:55:20.192539 17704 catalog_manager.cc:5495] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1). New cstate: current_term: 1 leader_uuid: "a2337128874c44b6ac7610cd44bf8f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } health_report { overall_health: UNKNOWN } } }
I20260619 16:55:20.214608 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.206929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:40935 (local address 127.0.0.1:60176)
0619 16:55:20.207209 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:20.207234 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:20.207391 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:20.208154 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:20.208159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:20.208174 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:55:20.208389 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.208404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.209598 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.209603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.211784 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.211806 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.212723 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.212729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.212892 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.212962 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:55:20.213964 (+ 1002us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:55:20.213992 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:55:20.214107 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 16:55:20.214274 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 16:55:20.214653 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.207060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.16.48.1:40935)
0619 16:55:20.207722 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.207726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.207746 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:55:20.207826 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.207829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.207906 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.208052 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:55:20.208564 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.209416 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.211978 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.212524 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.212701 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.213220 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:55:20.213845 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:55:20.214000 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.214149 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.214238 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":356,"threads_started":1}
I20260619 16:55:20.219513 18139 raft_consensus.cc:1261] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 1 FOLLOWER]: Refusing update from remote peer a2337128874c44b6ac7610cd44bf8f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:20.219852 17998 raft_consensus.cc:1261] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Refusing update from remote peer a2337128874c44b6ac7610cd44bf8f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:20.220323 18252 consensus_queue.cc:1035] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:20.220674 18215 consensus_queue.cc:1035] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:20.225337 18219 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: TxnStatusTablet state changed. Reason: New leader a2337128874c44b6ac7610cd44bf8f78. Latest consensus state: current_term: 1 leader_uuid: "a2337128874c44b6ac7610cd44bf8f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:20.225725 18219 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:20.232847 18258 mvcc.cc:205] Tried to move back new op lower bound from 7298613740411113472 to 7298613740296019968. Current Snapshot: MvccSnapshot[applied={T|T < 7298613740411113472}]
I20260619 16:55:20.236318 18215 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2337128874c44b6ac7610cd44bf8f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:20.236656 18215 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:55:20.240592 18219 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2337128874c44b6ac7610cd44bf8f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:20.240896 18219 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P 3cb756c0a4b84f7291e0292af91db279: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:20.241255 18216 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: TxnStatusTablet state changed. Reason: New leader a2337128874c44b6ac7610cd44bf8f78. Latest consensus state: current_term: 1 leader_uuid: "a2337128874c44b6ac7610cd44bf8f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:20.241452 18216 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:20.251523 18216 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2337128874c44b6ac7610cd44bf8f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:20.251796 18216 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:20.253978 18219 consensus_queue.cc:1035] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:20.254837 18252 tablet_replica.cc:439] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2337128874c44b6ac7610cd44bf8f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 } } peers { permanent_uuid: "3cb756c0a4b84f7291e0292af91db279" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41969 } } peers { permanent_uuid: "1e8a68e33b5a4b2393c8f2b7f278c4e0" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41557 } } }
I20260619 16:55:20.258663 18252 tablet_replica.cc:442] T 4849780340cc458cba119e10238d3565 P a2337128874c44b6ac7610cd44bf8f78: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:55:20.260139 18227 consensus_queue.cc:1035] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2337128874c44b6ac7610cd44bf8f78" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:20.263317 18259 mvcc.cc:205] Tried to move back new op lower bound from 7298613740411113472 to 7298613740296019968. Current Snapshot: MvccSnapshot[applied={T|T < 7298613740411113472}]
I20260619 16:55:20.290048 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.283391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:58857 (local address 127.16.48.1:40935)
0619 16:55:20.283582 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.283589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.284278 (+ 689us) server_negotiation.cc:407] Connection header received
0619 16:55:20.284493 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.284504 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.284574 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.284698 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:55:20.285171 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.286078 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.287649 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.288216 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.288366 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.289263 (+ 897us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:55:20.289386 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:55:20.289489 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.289619 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.289713 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:55:20.290105 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.283416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:40935 (local address 127.16.48.2:58857)
0619 16:55:20.284101 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:20.284181 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:20.284339 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:20.284713 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:20.284717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:20.284728 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:55:20.285020 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.285029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.286232 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.286238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.287490 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.287505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.288847 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.288855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.288987 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.289049 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:55:20.289490 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:55:20.289504 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 16:55:20.289589 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:55:20.289745 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":355,"threads_started":1}
I20260619 16:55:20.341333 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.334498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:49739 (local address 127.16.48.2:41969)
0619 16:55:20.334698 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.334705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.335113 (+ 408us) server_negotiation.cc:407] Connection header received
0619 16:55:20.335356 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.335361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.335455 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.335560 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:55:20.336248 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.336991 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.339322 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.339982 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.340142 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.340704 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:55:20.340811 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:55:20.340884 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.341041 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.341107 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:55:20.341630 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.333967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41969 (local address 127.16.48.1:49739)
0619 16:55:20.334979 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:20.335001 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:20.335166 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:20.335595 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:20.335600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:20.335614 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:55:20.336023 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.336035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.337147 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.337154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.339139 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:20.339163 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.340219 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.340227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:20.340420 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.340507 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:55:20.340916 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:55:20.340927 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 16:55:20.341016 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:55:20.341203 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":618,"threads_started":1}
I20260619 16:55:20.379382 17703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:20.379665 17703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:20.383080 17703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02951ec0faa94564bb6d617fe20b3aa2: Sending DeleteTablet for 3 replicas of tablet 07e87978df9c4e528a9dd3a859d78fe2
I20260619 16:55:20.384021 18119 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e87978df9c4e528a9dd3a859d78fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:20 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:59418
I20260619 16:55:20.384081 17979 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e87978df9c4e528a9dd3a859d78fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:20 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:47750
I20260619 16:55:20.384780 18277 tablet_replica.cc:330] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: stopping tablet replica
I20260619 16:55:20.384859 18278 tablet_replica.cc:330] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: stopping tablet replica
I20260619 16:55:20.384809 17840 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e87978df9c4e528a9dd3a859d78fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:20 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:60160
I20260619 16:55:20.384987 18277 raft_consensus.cc:2229] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:20.385025 18278 raft_consensus.cc:2229] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:55:20.385248 18277 raft_consensus.cc:2258] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:20.385457 18278 raft_consensus.cc:2258] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:20.386600 18278 ts_tablet_manager.cc:1905] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:20.387108 18277 ts_tablet_manager.cc:1905] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:20.394913 18278 ts_tablet_manager.cc:1918] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:55:20.394914 18277 ts_tablet_manager.cc:1918] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:55:20.395112 18278 log.cc:1196] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/07e87978df9c4e528a9dd3a859d78fe2
I20260619 16:55:20.395112 18277 log.cc:1196] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/07e87978df9c4e528a9dd3a859d78fe2
I20260619 16:55:20.395635 18277 ts_tablet_manager.cc:1939] T 07e87978df9c4e528a9dd3a859d78fe2 P 1e8a68e33b5a4b2393c8f2b7f278c4e0: Deleting consensus metadata
I20260619 16:55:20.395637 18278 ts_tablet_manager.cc:1939] T 07e87978df9c4e528a9dd3a859d78fe2 P 3cb756c0a4b84f7291e0292af91db279: Deleting consensus metadata
I20260619 16:55:20.396552 18279 tablet_replica.cc:330] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: stopping tablet replica
I20260619 16:55:20.396828 18279 raft_consensus.cc:2229] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:20.397046 17691 catalog_manager.cc:4841] TS 1e8a68e33b5a4b2393c8f2b7f278c4e0 (127.16.48.3:41557): tablet 07e87978df9c4e528a9dd3a859d78fe2 (table test-table [id=e0026d2f12e842fd8bf26bbcbbc90f48]) successfully deleted
I20260619 16:55:20.397167 18279 raft_consensus.cc:2258] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:20.398362 18279 ts_tablet_manager.cc:1905] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:20.404855 18279 ts_tablet_manager.cc:1918] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:55:20.405056 18279 log.cc:1196] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/07e87978df9c4e528a9dd3a859d78fe2
I20260619 16:55:20.405180 17688 catalog_manager.cc:4841] TS 3cb756c0a4b84f7291e0292af91db279 (127.16.48.2:41969): tablet 07e87978df9c4e528a9dd3a859d78fe2 (table test-table [id=e0026d2f12e842fd8bf26bbcbbc90f48]) successfully deleted
I20260619 16:55:20.405689 18279 ts_tablet_manager.cc:1939] T 07e87978df9c4e528a9dd3a859d78fe2 P a2337128874c44b6ac7610cd44bf8f78: Deleting consensus metadata
I20260619 16:55:20.406948 17691 catalog_manager.cc:4841] TS a2337128874c44b6ac7610cd44bf8f78 (127.16.48.1:40935): tablet 07e87978df9c4e528a9dd3a859d78fe2 (table test-table [id=e0026d2f12e842fd8bf26bbcbbc90f48]) successfully deleted
Jun 19 16:55:20 dist-test-slave-3ffd krb5kdc[17659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:20.424021 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.398905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.16.48.1:40935)
0619 16:55:20.399108 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.399114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.405672 (+ 6558us) server_negotiation.cc:407] Connection header received
0619 16:55:20.406143 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.406149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.406270 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.406419 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:20.408749 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.409572 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.411043 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.411462 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.411607 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.416246 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:20.416268 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:20.416287 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:20.416343 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:20.420013 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.421653 (+ 1640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.421663 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.421666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.421848 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.422302 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.422306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.422308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.422674 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:20.422915 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.423350 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.423598 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 16:55:20.424894 17906 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60182
I20260619 16:55:20.443893 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.428398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.16.48.62:40121)
0619 16:55:20.429468 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.429474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.429495 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:55:20.429558 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.429564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.429645 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.429761 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:20.431186 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.432320 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.433649 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.434400 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.434529 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.437247 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:20.437254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:20.437259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:20.437309 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:20.440965 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.441771 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.441778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.441782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.441955 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.442339 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.442344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.442348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.442717 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:20.442989 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.443265 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.443506 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":772,"threads_started":1}
W20260619 16:55:20.444630 17703 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57600
I20260619 16:55:20.446516 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 17775
I20260619 16:55:20.462925 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.448079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.16.48.1:40935)
0619 16:55:20.448318 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.448325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.448670 (+ 345us) server_negotiation.cc:407] Connection header received
0619 16:55:20.448827 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.448837 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.448938 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.449048 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:20.450232 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.451343 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.452629 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.453409 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.453543 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.455876 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:20.455883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:20.455889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:20.455932 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:20.459968 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.460818 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.460824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.460828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.460981 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.461354 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.461360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.461368 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.461771 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:20.462036 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.462280 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.462534 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260619 16:55:20.463613 17906 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60198
W20260619 16:55:20.464345 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 17775: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:55:20.464550 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 17775
I20260619 16:55:20.484913 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 17915
Jun 19 16:55:20 dist-test-slave-3ffd krb5kdc[17659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:55:20.499410 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.485699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.16.48.2:41969)
0619 16:55:20.485902 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.485906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.485924 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:55:20.486048 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.486052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.486114 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.486213 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:20.487403 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.488231 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.489465 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.489872 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.489974 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.494240 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:20.494255 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:20.494272 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:20.494310 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:20.497048 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.497681 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.497689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.497691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.497808 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.498119 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.498122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.498124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.498442 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:20.498656 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.498873 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.499034 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 16:55:20.500281 18045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47782
W20260619 16:55:20.500942 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 17915: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:55:20.501122 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 17915
I20260619 16:55:20.522555 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18054
Jun 19 16:55:20 dist-test-slave-3ffd krb5kdc[17659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:55:20.538432 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:20.523386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.16.48.3:41557)
0619 16:55:20.523716 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:20.523721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:20.523741 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:55:20.523799 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:20.523803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:20.523868 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:20.523961 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:20.525391 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.526191 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.527404 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:20.527869 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:20.527977 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:20.532489 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:20.532531 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:20.532548 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:20.532597 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:20.535829 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.536704 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.536713 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.536715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.536862 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:20.537197 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:20.537200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:20.537202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:20.537578 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:20.537795 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:20.538013 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 16:55:20.538193 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 16:55:20.539146 18184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59434
W20260619 16:55:20.539863 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18054: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:55:20.540009 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 18054
I20260619 16:55:20.557137 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 17672
I20260619 16:55:20.558344 17744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:20.674228 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 17672
2026-06-19T16:55:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4942 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:20 dist-test-slave-3ffd krb5kdc[18289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:20 dist-test-slave-3ffd krb5kdc[18289](info): set up 2 sockets
Jun 19 16:55:20 dist-test-slave-3ffd krb5kdc[18289](info): commencing operation
krb5kdc: starting...
W20260619 16:55:21.764775 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:21 dist-test-slave-3ffd krb5kdc[18289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:21.968945 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:40191
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:40191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:22.307883 18302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:22.308324 18302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:22.308494 18302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:22.315984 18302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:22.316207 18302 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:55:22.316318 18302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:22.316437 18302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:22.316533 18302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:22.316623 18302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:22.326931 18302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:40191
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:40191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:22.328446 18302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:22.329908 18302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:22.343230 18307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:22.343189 18310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:22.343269 18308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:22.344393 18302 server_base.cc:1034] running on GCE node
I20260619 16:55:22.345265 18302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:22.346812 18302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:22.348186 18302 hybrid_clock.cc:648] HybridClock initialized: now 1781888122348041 us; error 65 us; skew 500 ppm
I20260619 16:55:22.348645 18302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:22 dist-test-slave-3ffd krb5kdc[18289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888122, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:22.352061 18302 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:22.353802 18302 webserver.cc:460] Webserver started at http://127.16.48.62:37109/ using document root <none> and password file <none>
I20260619 16:55:22.354557 18302 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:22.354650 18302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:22.354916 18302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:22.358199 18302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "be52940b5bab450b8941f76b2dacf01a"
format_stamp: "Formatted at 2026-06-19 16:55:22 on dist-test-slave-3ffd"
server_key: "b229a53734faa0ef3ab8d714c9833c35"
server_key_iv: "6776386dc3fb71dfc80edee7879ca4f2"
server_key_version: "encryptionkey@0"
I20260619 16:55:22.359025 18302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "be52940b5bab450b8941f76b2dacf01a"
format_stamp: "Formatted at 2026-06-19 16:55:22 on dist-test-slave-3ffd"
server_key: "b229a53734faa0ef3ab8d714c9833c35"
server_key_iv: "6776386dc3fb71dfc80edee7879ca4f2"
server_key_version: "encryptionkey@0"
I20260619 16:55:22.365167 18302 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 16:55:22.369521 18317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:22.371275 18302 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260619 16:55:22.371531 18302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "be52940b5bab450b8941f76b2dacf01a"
format_stamp: "Formatted at 2026-06-19 16:55:22 on dist-test-slave-3ffd"
server_key: "b229a53734faa0ef3ab8d714c9833c35"
server_key_iv: "6776386dc3fb71dfc80edee7879ca4f2"
server_key_version: "encryptionkey@0"
I20260619 16:55:22.371810 18302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:22.398382 18302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:22.402976 18302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:22.403297 18302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:22.426340 18302 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:40191
I20260619 16:55:22.426358 18378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:40191 every 8 connection(s)
I20260619 16:55:22.427865 18302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:22.429211 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 18302
I20260619 16:55:22.429430 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:22.430081 16576 external_mini_cluster.cc:1428] Setting key 98038f1d1ed08ac51092fd3ee3a9161f
I20260619 16:55:22.432880 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:55:22 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:22.446663 18379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a: Bootstrap starting.
I20260619 16:55:22.451814 18379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:22.453356 18379 log.cc:824] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:22.457427 18379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a: No bootstrap required, opened a new log
I20260619 16:55:22.457614 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:22.432243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.16.48.62:40191)
0619 16:55:22.433043 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:22.433059 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:22.433113 (+ 54us) server_negotiation.cc:407] Connection header received
0619 16:55:22.435429 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:22.435461 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:22.436032 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:22.436591 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:22.437960 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:22.440167 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:22.441514 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:22.442633 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:22.442863 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:22.447231 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:22.447244 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:22.447270 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:22.447334 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:22.451885 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:22.452862 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:22.452873 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:22.452884 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:22.453165 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:22.453769 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:22.453775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:22.453778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:22.454593 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:22.454896 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:22.455489 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 16:55:22.456020 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":325,"threads_started":1}
I20260619 16:55:22.465880 18379 raft_consensus.cc:348] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be52940b5bab450b8941f76b2dacf01a" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40191 } }
I20260619 16:55:22.466324 18379 raft_consensus.cc:374] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:22.466416 18379 raft_consensus.cc:729] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be52940b5bab450b8941f76b2dacf01a, State: Initialized, Role: FOLLOWER
I20260619 16:55:22.467092 18379 consensus_queue.cc:260] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be52940b5bab450b8941f76b2dacf01a" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40191 } }
I20260619 16:55:22.467320 18379 raft_consensus.cc:388] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:22.467434 18379 raft_consensus.cc:482] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:22.467582 18379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:22.470155 18379 raft_consensus.cc:504] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be52940b5bab450b8941f76b2dacf01a" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40191 } }
I20260619 16:55:22.470784 18379 leader_election.cc:302] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be52940b5bab450b8941f76b2dacf01a; no voters:
I20260619 16:55:22.471432 18379 leader_election.cc:288] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:22.471554 18384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:22.472409 18384 raft_consensus.cc:686] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [term 1 LEADER]: Becoming Leader. State: Replica: be52940b5bab450b8941f76b2dacf01a, State: Running, Role: LEADER
I20260619 16:55:22.473067 18384 consensus_queue.cc:237] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be52940b5bab450b8941f76b2dacf01a" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40191 } }
I20260619 16:55:22.473845 18379 sys_catalog.cc:564] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:22.477787 18386 sys_catalog.cc:455] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [sys.catalog]: SysCatalogTable state changed. Reason: New leader be52940b5bab450b8941f76b2dacf01a. Latest consensus state: current_term: 1 leader_uuid: "be52940b5bab450b8941f76b2dacf01a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be52940b5bab450b8941f76b2dacf01a" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40191 } } }
I20260619 16:55:22.478469 18386 sys_catalog.cc:458] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:22.479818 18385 sys_catalog.cc:455] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be52940b5bab450b8941f76b2dacf01a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be52940b5bab450b8941f76b2dacf01a" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40191 } } }
I20260619 16:55:22.480453 18385 sys_catalog.cc:458] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:22.482949 18392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:22.490657 18392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:22.501400 18392 catalog_manager.cc:1347] Generated new cluster ID: 9f65c6b514ca4aae8eeba1390a0ede69
I20260619 16:55:22.501573 18392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:22.514999 18392 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:22.516693 18392 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:22.528074 18392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a: Generated new TSK 0
I20260619 16:55:22.529178 18392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:55:22.592938 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:40191
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:22.956105 18405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:22.956509 18405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:22.956691 18405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:22.964511 18405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:22.964686 18405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:22.964828 18405 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:55:22.964915 18405 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:55:22.965046 18405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:22.975781 18405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.48.62:40191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:22.977380 18405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:22.978881 18405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:22.993618 18410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:22.993659 18411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:22.993836 18413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:22.994628 18405 server_base.cc:1034] running on GCE node
I20260619 16:55:22.995368 18405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:22.996543 18405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:22.997830 18405 hybrid_clock.cc:648] HybridClock initialized: now 1781888122997774 us; error 55 us; skew 500 ppm
I20260619 16:55:22.998200 18405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:22 dist-test-slave-3ffd krb5kdc[18289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888122, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:23.001645 18405 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:55:23.003283 18405 webserver.cc:460] Webserver started at http://127.16.48.1:33659/ using document root <none> and password file <none>
I20260619 16:55:23.004065 18405 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:23.004251 18405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:23.004594 18405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:23.007702 18405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "dbf05a8c520447da9f3aa3c9cdb1e924"
format_stamp: "Formatted at 2026-06-19 16:55:23 on dist-test-slave-3ffd"
server_key: "10aa20c735a58f8913deaa257a90a059"
server_key_iv: "cbe26f9a4aef0bf3668f0026b9827b04"
server_key_version: "encryptionkey@0"
I20260619 16:55:23.008582 18405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "dbf05a8c520447da9f3aa3c9cdb1e924"
format_stamp: "Formatted at 2026-06-19 16:55:23 on dist-test-slave-3ffd"
server_key: "10aa20c735a58f8913deaa257a90a059"
server_key_iv: "cbe26f9a4aef0bf3668f0026b9827b04"
server_key_version: "encryptionkey@0"
I20260619 16:55:23.014282 18405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:55:23.018481 18420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:23.020071 18405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:55:23.020340 18405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "dbf05a8c520447da9f3aa3c9cdb1e924"
format_stamp: "Formatted at 2026-06-19 16:55:23 on dist-test-slave-3ffd"
server_key: "10aa20c735a58f8913deaa257a90a059"
server_key_iv: "cbe26f9a4aef0bf3668f0026b9827b04"
server_key_version: "encryptionkey@0"
I20260619 16:55:23.020577 18405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:23.039986 18405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:23.044356 18405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:23.044690 18405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:23.047499 18405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:23.047650 18405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:23.047811 18405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:23.047910 18405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:55:23 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888122, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:23.074844 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:23.048430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:49371 (local address 127.16.48.62:40191)
0619 16:55:23.048896 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:23.048903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:23.050315 (+ 1412us) server_negotiation.cc:407] Connection header received
0619 16:55:23.053742 (+ 3427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:23.053751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:23.053865 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:23.054011 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:23.057136 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.058144 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.060214 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.060845 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.060999 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:23.066508 (+ 5509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:23.066519 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:23.066526 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:23.066589 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:23.070444 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:23.071654 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:23.071665 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:23.071670 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:23.071855 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:23.072642 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:23.072646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:23.072648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:23.073025 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:23.073195 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:23.074190 (+ 995us) server_negotiation.cc:299] Negotiation successful
0619 16:55:23.074433 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":225,"threads_started":1}
I20260619 16:55:23.076054 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:23.049140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40191 (local address 127.16.48.1:49371)
0619 16:55:23.050088 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:23.050159 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:23.053220 (+ 3061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:23.054392 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:23.054411 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:23.055535 (+ 1124us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:23.056843 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:23.056883 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.058356 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.058362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:23.059990 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:23.060008 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.061131 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.061139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:23.061518 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:23.062463 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:23.062508 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:23.066116 (+ 3608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:23.070637 (+ 4521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:23.070648 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:23.070676 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:23.071442 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:23.072025 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:23.072030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:23.072034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:23.072455 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:23.073273 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:23.073281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:23.073768 (+ 487us) client_negotiation.cc:769] Sending connection context
0619 16:55:23.074064 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 16:55:23.074662 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":372,"threads_started":1}
I20260619 16:55:23.108743 18405 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:37037
I20260619 16:55:23.108822 18540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:37037 every 8 connection(s)
I20260619 16:55:23.110662 18405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:23.113397 18541 heartbeater.cc:347] Connected to a master server at 127.16.48.62:40191
I20260619 16:55:23.113675 18541 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:23.114638 18541 heartbeater.cc:511] Master 127.16.48.62:40191 requested a full tablet report, sending...
I20260619 16:55:23.115876 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:23.094376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:54493 (local address 127.16.48.62:40191)
0619 16:55:23.094620 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:23.094630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:23.097036 (+ 2406us) server_negotiation.cc:407] Connection header received
0619 16:55:23.097461 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:23.097469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:23.097568 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:23.097726 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:23.099453 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.100587 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.104175 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.105059 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.105308 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:23.108876 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:23.108884 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:23.108888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:23.108936 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:23.112611 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:23.113637 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:23.113641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:23.113644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:23.113821 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:23.114268 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:23.114274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:23.114277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:23.114584 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:23.114752 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:23.115263 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 16:55:23.115500 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:55:23.116273 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:23.094365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40191 (local address 127.16.48.1:54493)
0619 16:55:23.096864 (+ 2499us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:23.096895 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:23.097224 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:23.097793 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:23.097802 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:23.098634 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:23.099159 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:23.099180 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.100799 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.100810 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:23.103798 (+ 2988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:23.103829 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.105711 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.105720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:23.105987 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:23.106999 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:23.107022 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:23.108546 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:23.112795 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:23.112801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:23.112805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:23.113432 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:23.113952 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:23.113959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:23.113963 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:23.114152 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:23.114814 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:23.114823 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:23.114992 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 16:55:23.115151 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 16:55:23.115509 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2274,"thread_start_us":2047,"threads_started":1}
I20260619 16:55:23.116623 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 18405
I20260619 16:55:23.116782 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:23.117396 18334 ts_manager.cc:194] Registered new tserver with Master: dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1:37037)
I20260619 16:55:23.117601 16576 external_mini_cluster.cc:1428] Setting key 3a800aed1f8fa5a339f4800f50ba8a73
I20260619 16:55:23.119971 18334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:49371
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:55:23.169546 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:40191
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:23.510743 18545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:23.511046 18545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:23.511241 18545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:23.519057 18545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:23.519222 18545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:23.519392 18545 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:55:23.519474 18545 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:55:23.519605 18545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:23.529875 18545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.48.62:40191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:23.531306 18545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:23.532778 18545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:23.546941 18550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:23.546976 18551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:23.546905 18553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:23.547745 18545 server_base.cc:1034] running on GCE node
I20260619 16:55:23.548460 18545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:23.549430 18545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:23.550640 18545 hybrid_clock.cc:648] HybridClock initialized: now 1781888123550572 us; error 64 us; skew 500 ppm
I20260619 16:55:23.551013 18545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:23 dist-test-slave-3ffd krb5kdc[18289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888123, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:23.554394 18545 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:55:23.556085 18545 webserver.cc:460] Webserver started at http://127.16.48.2:45991/ using document root <none> and password file <none>
I20260619 16:55:23.556893 18545 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:23.557047 18545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:23.557394 18545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:23.560676 18545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "2708626cbf3d4bd29ad2078dcac27eeb"
format_stamp: "Formatted at 2026-06-19 16:55:23 on dist-test-slave-3ffd"
server_key: "43026dd1ac5b203f902057d864313c4a"
server_key_iv: "0493b773e5e582fe93d9b8a2996a7010"
server_key_version: "encryptionkey@0"
I20260619 16:55:23.561523 18545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "2708626cbf3d4bd29ad2078dcac27eeb"
format_stamp: "Formatted at 2026-06-19 16:55:23 on dist-test-slave-3ffd"
server_key: "43026dd1ac5b203f902057d864313c4a"
server_key_iv: "0493b773e5e582fe93d9b8a2996a7010"
server_key_version: "encryptionkey@0"
I20260619 16:55:23.567392 18545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:55:23.571453 18560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:23.573320 18545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:23.573551 18545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "2708626cbf3d4bd29ad2078dcac27eeb"
format_stamp: "Formatted at 2026-06-19 16:55:23 on dist-test-slave-3ffd"
server_key: "43026dd1ac5b203f902057d864313c4a"
server_key_iv: "0493b773e5e582fe93d9b8a2996a7010"
server_key_version: "encryptionkey@0"
I20260619 16:55:23.573793 18545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:23.591784 18545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:23.596766 18545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:23.597096 18545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:23.600020 18545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:23.600204 18545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:23.600348 18545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:23.600451 18545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:55:23 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888123, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:23.627167 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:23.600666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:55775 (local address 127.16.48.62:40191)
0619 16:55:23.600864 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:23.600870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:23.602504 (+ 1634us) server_negotiation.cc:407] Connection header received
0619 16:55:23.605038 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:23.605050 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:23.605179 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:23.605326 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:23.608025 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.609432 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.612349 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.613012 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.613180 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:23.618653 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:23.618662 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:23.618667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:23.618753 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:23.622099 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:23.623621 (+ 1522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:23.623631 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:23.623636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:23.623819 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:23.624613 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:23.624618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:23.624620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:23.624955 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:23.625101 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:23.626389 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0619 16:55:23.626718 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:55:23.629662 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:23.601335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40191 (local address 127.16.48.2:55775)
0619 16:55:23.602278 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:23.602333 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:23.604514 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:23.605639 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:23.605648 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:23.606578 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:23.607713 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:23.607746 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.609708 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.609716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:23.611993 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:23.612022 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.613305 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.613313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:23.613640 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:23.614518 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:23.614542 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:23.618267 (+ 3725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:23.622324 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:23.622360 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:23.622389 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:23.623360 (+ 971us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:23.623956 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:23.623964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:23.623970 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:23.624336 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:23.625102 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:23.625115 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:23.625781 (+ 666us) client_negotiation.cc:769] Sending connection context
0619 16:55:23.626164 (+ 383us) client_negotiation.cc:240] Negotiation successful
0619 16:55:23.626982 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":414,"threads_started":1}
I20260619 16:55:23.653815 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:23.636072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:33073 (local address 127.16.48.62:40191)
0619 16:55:23.636334 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:23.636343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:23.637060 (+ 717us) server_negotiation.cc:407] Connection header received
0619 16:55:23.637342 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:23.637346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:23.637415 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:23.637512 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:23.639091 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.640013 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.642773 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.643198 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.643310 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:23.646488 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:23.646493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:23.646497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:23.646533 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:23.650173 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:23.651373 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:23.651378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:23.651380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:23.651531 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:23.652016 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:23.652019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:23.652022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:23.652457 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:23.652622 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:23.653067 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 16:55:23.653318 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 16:55:23.653965 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:23.636021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40191 (local address 127.16.48.2:33073)
0619 16:55:23.636896 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:23.636930 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:23.637159 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:23.637586 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:23.637594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:23.638384 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:23.638873 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:23.638891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.640236 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.640249 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:23.642550 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:23.642581 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:23.643449 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:23.643459 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:23.643752 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:23.644755 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:23.644789 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:23.646236 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:23.650352 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:23.650362 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:23.650370 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:23.651191 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:23.651661 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:23.651669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:23.651676 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:23.651885 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:23.652620 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:23.652625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:23.652817 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 16:55:23.652966 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 16:55:23.653360 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":467,"threads_started":1}
I20260619 16:55:23.666923 18545 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:39389
I20260619 16:55:23.666960 18679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:39389 every 8 connection(s)
I20260619 16:55:23.668514 18545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:23.670770 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 18545
I20260619 16:55:23.670975 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:23.671044 18680 heartbeater.cc:347] Connected to a master server at 127.16.48.62:40191
I20260619 16:55:23.671295 18680 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:23.671551 16576 external_mini_cluster.cc:1428] Setting key 692847fb86710a15ba0a7df24e1b1660
I20260619 16:55:23.672204 18680 heartbeater.cc:511] Master 127.16.48.62:40191 requested a full tablet report, sending...
I20260619 16:55:23.673614 18334 ts_manager.cc:194] Registered new tserver with Master: 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389)
I20260619 16:55:23.674464 18334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:55775
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:55:23.717571 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:40191
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:24.064724 18684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:24.065039 18684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:24.065264 18684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:24.072769 18684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:24.072902 18684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:24.073086 18684 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:55:24.073172 18684 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:55:24.073308 18684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:24.084254 18684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.48.62:40191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:24.085731 18684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:24.087173 18684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:24.101593 18689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:24.101593 18690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:24.101809 18692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:24.102615 18684 server_base.cc:1034] running on GCE node
I20260619 16:55:24.103335 18684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:24.104452 18684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:24.105669 18684 hybrid_clock.cc:648] HybridClock initialized: now 1781888124105603 us; error 56 us; skew 500 ppm
I20260619 16:55:24.106046 18684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:24 dist-test-slave-3ffd krb5kdc[18289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888124, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:24.109968 18684 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:55:24.114522 18684 webserver.cc:460] Webserver started at http://127.16.48.3:42691/ using document root <none> and password file <none>
I20260619 16:55:24.115298 18684 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:24.115460 18684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:24.115818 18684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:24.119005 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "bd8485898450436b8467a7e5a723de7b"
format_stamp: "Formatted at 2026-06-19 16:55:24 on dist-test-slave-3ffd"
server_key: "2614a3b1ae487232593a55a97962fa84"
server_key_iv: "eced55e27f5463d694dec6b3353bab50"
server_key_version: "encryptionkey@0"
I20260619 16:55:24.119834 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "bd8485898450436b8467a7e5a723de7b"
format_stamp: "Formatted at 2026-06-19 16:55:24 on dist-test-slave-3ffd"
server_key: "2614a3b1ae487232593a55a97962fa84"
server_key_iv: "eced55e27f5463d694dec6b3353bab50"
server_key_version: "encryptionkey@0"
I20260619 16:55:24.123991 18541 heartbeater.cc:503] Master 127.16.48.62:40191 was elected leader, sending a full tablet report...
I20260619 16:55:24.125844 18684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:55:24.129760 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:24.131490 18684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:55:24.131687 18684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "bd8485898450436b8467a7e5a723de7b"
format_stamp: "Formatted at 2026-06-19 16:55:24 on dist-test-slave-3ffd"
server_key: "2614a3b1ae487232593a55a97962fa84"
server_key_iv: "eced55e27f5463d694dec6b3353bab50"
server_key_version: "encryptionkey@0"
I20260619 16:55:24.131922 18684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:24.146294 18684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:24.150921 18684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:24.151258 18684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:24.154433 18684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:24.154549 18684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:24.154702 18684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:24.154834 18684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:55:24 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888124, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:24.186939 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.155004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:47065 (local address 127.16.48.62:40191)
0619 16:55:24.158302 (+ 3298us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:24.158307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:24.158328 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:55:24.160911 (+ 2583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:24.160919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:24.161031 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:24.161136 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:24.164999 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.166104 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.170108 (+ 4004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.170731 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.170927 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.177047 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:24.177058 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:24.177067 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:24.177133 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:24.181921 (+ 4788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:24.183221 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:24.183233 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:24.183240 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:24.183473 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:24.184177 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:24.184186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:24.184192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:24.184684 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:24.184905 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:24.186182 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0619 16:55:24.186461 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3178,"thread_start_us":3070,"threads_started":1}
I20260619 16:55:24.188767 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.155915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40191 (local address 127.16.48.3:47065)
0619 16:55:24.157199 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:24.157284 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:24.160377 (+ 3093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:24.161951 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:24.161967 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:24.163168 (+ 1201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:24.164655 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.164697 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.166350 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.166356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.169718 (+ 3362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.169755 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.171052 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.171063 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.171363 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.172334 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:24.172364 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:24.176498 (+ 4134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:24.182189 (+ 5691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:24.182212 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:24.182241 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:24.183004 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:24.183668 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:24.183674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:24.183678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:24.184000 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:24.184882 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:24.184897 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:24.185571 (+ 674us) client_negotiation.cc:769] Sending connection context
0619 16:55:24.186061 (+ 490us) client_negotiation.cc:240] Negotiation successful
0619 16:55:24.186753 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":482,"threads_started":1}
I20260619 16:55:24.213153 18684 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:39015
I20260619 16:55:24.213186 18819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:39015 every 8 connection(s)
I20260619 16:55:24.214973 18684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:24.217346 18820 heartbeater.cc:347] Connected to a master server at 127.16.48.62:40191
I20260619 16:55:24.217625 18820 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:24.217891 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.195623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:35291 (local address 127.16.48.62:40191)
0619 16:55:24.195795 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:24.195799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:24.196654 (+ 855us) server_negotiation.cc:407] Connection header received
0619 16:55:24.196918 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:24.196923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:24.196989 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:24.197106 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:24.198083 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.198789 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.200169 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.200701 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.200866 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.210479 (+ 9613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:24.210491 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:24.210499 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:24.210564 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:24.214466 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:24.215357 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:24.215364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:24.215368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:24.215558 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:24.216079 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:24.216085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:24.216088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:24.216515 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:24.216659 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:24.217271 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 16:55:24.217517 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:55:24.218132 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.195666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40191 (local address 127.16.48.3:35291)
0619 16:55:24.196517 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:24.196549 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:24.196747 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:24.197131 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:24.197135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:24.197638 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:24.197928 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.197938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.198950 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.198955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.199983 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.199994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.200951 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.200961 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.201265 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.208439 (+ 7174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:24.208482 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:24.210034 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:24.214679 (+ 4645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:24.214684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:24.214687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:24.215214 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:24.215716 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:24.215724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:24.215734 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:24.215906 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:24.216759 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:24.216762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:24.216972 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 16:55:24.217146 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 16:55:24.217558 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":446,"threads_started":1}
I20260619 16:55:24.218484 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 18684
I20260619 16:55:24.218668 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:24.218792 18820 heartbeater.cc:511] Master 127.16.48.62:40191 requested a full tablet report, sending...
I20260619 16:55:24.219235 16576 external_mini_cluster.cc:1428] Setting key 0c3e899b8462581873107f835348d0ae
I20260619 16:55:24.220470 18334 ts_manager.cc:194] Registered new tserver with Master: bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015)
I20260619 16:55:24.221630 18334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:47065
I20260619 16:55:24.223012 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:55:24 dist-test-slave-3ffd krb5kdc[18289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:55:24 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:24.272099 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.258096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.16.48.62:40191)
0619 16:55:24.258352 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:24.258360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:24.258536 (+ 176us) server_negotiation.cc:407] Connection header received
0619 16:55:24.258691 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:24.258695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:24.258804 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:24.258895 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:24.260445 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.261148 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.262379 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.262794 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.262891 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.266915 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:24.266921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:24.266924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:24.266966 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:24.269792 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:24.270416 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:24.270420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:24.270422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:24.270551 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:24.270869 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:24.270872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:24.270874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:24.271150 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:24.271340 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:24.271577 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:55:24.271760 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 16:55:24.276535 18333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:55:24.283645 18333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:24.307992 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.297114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:39389 (local address 127.0.0.1:60052)
0619 16:55:24.299411 (+ 2297us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:24.299430 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:24.299628 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:24.300581 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:24.300586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:24.300609 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:24.300995 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.301002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.303166 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.303176 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.305147 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.305159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.306929 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.306935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.307144 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.307204 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 16:55:24.307320 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 16:55:24.307530 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2168,"mutex_wait_us":23,"thread_start_us":2037,"threads_started":1}
I20260619 16:55:24.308663 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.298561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:37037 (local address 127.0.0.1:46856)
0619 16:55:24.301555 (+ 2994us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:24.301573 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:24.301695 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:24.302274 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:24.302278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:24.302300 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:24.302659 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.302670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.304269 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.304276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.306164 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.306174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.307816 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.307823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.308017 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.308060 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 16:55:24.308259 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 16:55:24.308392 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2866,"thread_start_us":2732,"threads_started":1}
I20260619 16:55:24.309371 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.297517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.16.48.2:39389)
0619 16:55:24.298512 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:24.298521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:24.299523 (+ 1002us) server_negotiation.cc:407] Connection header received
0619 16:55:24.299822 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:24.299830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:24.300076 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:24.300306 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:24.301156 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.303012 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.305332 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.306699 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.308585 (+ 1886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.308802 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:24.308884 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 16:55:24.309069 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":1722,"threads_started":1}
I20260619 16:55:24.309383 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.296952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:39015 (local address 127.0.0.1:51758)
0619 16:55:24.297742 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:24.297788 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:24.297973 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:24.299824 (+ 1851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:24.299836 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:24.299877 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:24.300356 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.300372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.302726 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.302734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.305491 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.305505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.308146 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.308157 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.308372 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.308402 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:55:24.308467 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 16:55:24.308612 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":54,"thread_start_us":713,"threads_started":1}
I20260619 16:55:24.309430 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.299085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.16.48.1:37037)
0619 16:55:24.300662 (+ 1577us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:24.300672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:24.301693 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 16:55:24.301829 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:24.301836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:24.302061 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:24.302275 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:24.302849 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.304060 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.306329 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.307593 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.307756 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.307993 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:24.308943 (+ 950us) server_negotiation.cc:299] Negotiation successful
0619 16:55:24.309142 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1409,"thread_start_us":1294,"threads_started":1}
I20260619 16:55:24.309511 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.297249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.16.48.3:39015)
0619 16:55:24.299058 (+ 1809us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:24.299068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:24.299103 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:55:24.299205 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:24.299215 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:24.299519 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:24.300583 (+ 1064us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:24.300995 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.302590 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.305681 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.307857 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.308758 (+ 901us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.308976 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:24.309038 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 16:55:24.309192 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1634,"thread_start_us":1451,"threads_started":1}
I20260619 16:55:24.313700 18610 tablet_service.cc:1467] Processing CreateTablet for tablet df4b6743774f462ebe3a472343de15bd (DEFAULT_TABLE table=test-table [id=7a99d33c10f64cc1a4794c662c5c7dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:24.313727 18471 tablet_service.cc:1467] Processing CreateTablet for tablet df4b6743774f462ebe3a472343de15bd (DEFAULT_TABLE table=test-table [id=7a99d33c10f64cc1a4794c662c5c7dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:24.313931 18750 tablet_service.cc:1467] Processing CreateTablet for tablet df4b6743774f462ebe3a472343de15bd (DEFAULT_TABLE table=test-table [id=7a99d33c10f64cc1a4794c662c5c7dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:24.315078 18610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4b6743774f462ebe3a472343de15bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:24.315140 18471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4b6743774f462ebe3a472343de15bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:24.315155 18750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4b6743774f462ebe3a472343de15bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:24.327713 18839 tablet_bootstrap.cc:492] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: Bootstrap starting.
I20260619 16:55:24.331717 18840 tablet_bootstrap.cc:492] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: Bootstrap starting.
I20260619 16:55:24.332685 18839 tablet_bootstrap.cc:654] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:24.334293 18839 log.cc:824] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:24.335693 18841 tablet_bootstrap.cc:492] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: Bootstrap starting.
I20260619 16:55:24.338305 18839 tablet_bootstrap.cc:492] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: No bootstrap required, opened a new log
I20260619 16:55:24.338658 18839 ts_tablet_manager.cc:1397] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260619 16:55:24.339849 18841 tablet_bootstrap.cc:654] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:24.339874 18840 tablet_bootstrap.cc:654] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:24.341284 18840 log.cc:824] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:24.341286 18841 log.cc:824] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:24.344772 18841 tablet_bootstrap.cc:492] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: No bootstrap required, opened a new log
I20260619 16:55:24.344873 18840 tablet_bootstrap.cc:492] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: No bootstrap required, opened a new log
I20260619 16:55:24.345113 18841 ts_tablet_manager.cc:1397] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 16:55:24.345172 18840 ts_tablet_manager.cc:1397] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20260619 16:55:24.350090 18839 raft_consensus.cc:348] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } }
I20260619 16:55:24.350642 18839 raft_consensus.cc:374] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:24.350742 18839 raft_consensus.cc:729] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708626cbf3d4bd29ad2078dcac27eeb, State: Initialized, Role: FOLLOWER
I20260619 16:55:24.351557 18839 consensus_queue.cc:260] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } }
I20260619 16:55:24.352880 18680 heartbeater.cc:503] Master 127.16.48.62:40191 was elected leader, sending a full tablet report...
I20260619 16:55:24.353636 18839 ts_tablet_manager.cc:1428] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260619 16:55:24.354094 18841 raft_consensus.cc:348] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } }
I20260619 16:55:24.354566 18841 raft_consensus.cc:374] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:24.354674 18841 raft_consensus.cc:729] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8485898450436b8467a7e5a723de7b, State: Initialized, Role: FOLLOWER
I20260619 16:55:24.355352 18841 consensus_queue.cc:260] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } }
I20260619 16:55:24.356684 18820 heartbeater.cc:503] Master 127.16.48.62:40191 was elected leader, sending a full tablet report...
I20260619 16:55:24.357290 18841 ts_tablet_manager.cc:1428] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 16:55:24.356209 18840 raft_consensus.cc:348] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } }
I20260619 16:55:24.357792 18840 raft_consensus.cc:374] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:24.357908 18840 raft_consensus.cc:729] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Initialized, Role: FOLLOWER
I20260619 16:55:24.358542 18840 consensus_queue.cc:260] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } }
I20260619 16:55:24.360543 18840 ts_tablet_manager.cc:1428] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260619 16:55:24.364248 18542 tablet.cc:2369] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:55:24.420929 18681 tablet.cc:2369] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:55:24.466945 18821 tablet.cc:2369] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:55:24.743003 18847 raft_consensus.cc:482] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:24.743279 18847 raft_consensus.cc:504] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } }
I20260619 16:55:24.745087 18847 leader_election.cc:288] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015), 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389)
I20260619 16:55:24.752063 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.745913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:39909 (local address 127.16.48.2:39389)
0619 16:55:24.746103 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:24.746111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:24.746482 (+ 371us) server_negotiation.cc:407] Connection header received
0619 16:55:24.746720 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:24.746724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:24.746786 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:24.746888 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:24.747469 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.748272 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.750187 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.750986 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.751144 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.751295 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:24.751638 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 16:55:24.751797 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:55:24.752051 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.745485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:55587 (local address 127.16.48.3:39015)
0619 16:55:24.745706 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:24.745713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:24.746602 (+ 889us) server_negotiation.cc:407] Connection header received
0619 16:55:24.746812 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:24.746816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:24.746875 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:24.746974 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:24.747619 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.748398 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.750226 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.750992 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.751159 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.751283 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:24.751634 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 16:55:24.751767 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:55:24.752410 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.745384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:39389 (local address 127.16.48.1:39909)
0619 16:55:24.746335 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:24.746357 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:24.746538 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:24.746901 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:24.746905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:24.746928 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:24.747308 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.747318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.748402 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.748410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.750012 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.750028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.751300 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.751309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.751460 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.751502 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:55:24.751610 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 16:55:24.751760 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":22,"thread_start_us":504,"threads_started":1}
I20260619 16:55:24.752559 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.745541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:39015 (local address 127.16.48.1:55587)
0619 16:55:24.746475 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:24.746495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:24.746671 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:24.746995 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:24.746999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:24.747047 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:24.747458 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.747467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.748545 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.748550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.750062 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.750074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.751247 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.751253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.751412 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.751493 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 16:55:24.751609 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 16:55:24.751761 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":89,"thread_start_us":656,"threads_started":1}
I20260619 16:55:24.753466 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4b6743774f462ebe3a472343de15bd" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b" is_pre_election: true
I20260619 16:55:24.753476 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4b6743774f462ebe3a472343de15bd" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" is_pre_election: true
I20260619 16:55:24.753932 18770 raft_consensus.cc:2454] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:24.753935 18635 raft_consensus.cc:2454] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:24.754596 18421 leader_election.cc:302] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2708626cbf3d4bd29ad2078dcac27eeb, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:24.755080 18847 raft_consensus.cc:2781] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:24.755244 18847 raft_consensus.cc:482] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:24.755353 18847 raft_consensus.cc:3037] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:24.757953 18847 raft_consensus.cc:504] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } }
I20260619 16:55:24.758625 18847 leader_election.cc:288] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Requested vote from peers bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015), 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389)
I20260619 16:55:24.759387 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4b6743774f462ebe3a472343de15bd" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b"
I20260619 16:55:24.759363 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4b6743774f462ebe3a472343de15bd" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb"
I20260619 16:55:24.759613 18770 raft_consensus.cc:3037] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:24.759613 18635 raft_consensus.cc:3037] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:24.761960 18635 raft_consensus.cc:2454] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:24.762492 18421 leader_election.cc:302] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2708626cbf3d4bd29ad2078dcac27eeb, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:24.762864 18847 raft_consensus.cc:2781] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:24.763338 18770 raft_consensus.cc:2454] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:24.763407 18847 raft_consensus.cc:686] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 LEADER]: Becoming Leader. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Running, Role: LEADER
I20260619 16:55:24.763989 18847 consensus_queue.cc:237] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } }
I20260619 16:55:24.769022 18334 catalog_manager.cc:5495] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1). New cstate: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } health_report { overall_health: HEALTHY } } }
Jun 19 16:55:24 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888122, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:24.908821 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.894655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.16.48.62:40191)
0619 16:55:24.895435 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:24.895442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:24.895635 (+ 193us) server_negotiation.cc:407] Connection header received
0619 16:55:24.895854 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:24.895859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:24.895922 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:24.896033 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:24.897412 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.898196 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.899453 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.899879 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.900025 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.903497 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:24.903502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:24.903505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:24.903534 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:24.906055 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:24.906789 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:24.906796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:24.906806 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:24.906927 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:24.907275 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:24.907283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:24.907287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:24.907555 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:24.907714 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:24.908106 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 16:55:24.908404 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":591,"thread_start_us":409,"threads_started":1}
I20260619 16:55:24.908897 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:24.894655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:40191 (local address 127.0.0.1:42604)
0619 16:55:24.895516 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:24.895532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:24.895722 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:24.896062 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:24.896065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:24.896979 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:24.897239 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.897251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.898336 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.898343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.899305 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:24.899317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:24.900049 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:24.900054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:24.900307 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:24.900999 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:24.901019 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:24.903277 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:24.906209 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:24.906227 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:24.906235 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:24.906668 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:24.907033 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:24.907040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:24.907043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:24.907174 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:24.907689 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:24.907697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:24.907866 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 16:55:24.908034 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 16:55:24.908524 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":167,"thread_start_us":424,"threads_started":1}
I20260619 16:55:24.914211 18334 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:42604:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:55:24.915340 18334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:24.922776 18750 tablet_service.cc:1467] Processing CreateTablet for tablet 9f157f6a93c44b8ab112f27c94feecf3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9a611a08f6046398940758d03685a7f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:55:24.922675 18610 tablet_service.cc:1467] Processing CreateTablet for tablet 9f157f6a93c44b8ab112f27c94feecf3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9a611a08f6046398940758d03685a7f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:55:24.922614 18471 tablet_service.cc:1467] Processing CreateTablet for tablet 9f157f6a93c44b8ab112f27c94feecf3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9a611a08f6046398940758d03685a7f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:55:24.923658 18610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f157f6a93c44b8ab112f27c94feecf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:24.923655 18750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f157f6a93c44b8ab112f27c94feecf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:24.923655 18471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f157f6a93c44b8ab112f27c94feecf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:24.932422 18859 tablet_bootstrap.cc:492] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: Bootstrap starting.
I20260619 16:55:24.932744 18861 tablet_bootstrap.cc:492] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: Bootstrap starting.
I20260619 16:55:24.934149 18861 tablet_bootstrap.cc:654] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:24.934876 18859 tablet_bootstrap.cc:654] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:24.935485 18860 tablet_bootstrap.cc:492] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: Bootstrap starting.
I20260619 16:55:24.936964 18861 tablet_bootstrap.cc:492] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: No bootstrap required, opened a new log
I20260619 16:55:24.937171 18861 ts_tablet_manager.cc:1397] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 16:55:24.937366 18860 tablet_bootstrap.cc:654] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:24.938261 18861 raft_consensus.cc:348] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:24.938546 18861 raft_consensus.cc:374] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:24.938645 18861 raft_consensus.cc:729] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8485898450436b8467a7e5a723de7b, State: Initialized, Role: FOLLOWER
I20260619 16:55:24.938953 18861 consensus_queue.cc:260] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:24.939550 18860 tablet_bootstrap.cc:492] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: No bootstrap required, opened a new log
I20260619 16:55:24.939646 18859 tablet_bootstrap.cc:492] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: No bootstrap required, opened a new log
I20260619 16:55:24.939721 18860 ts_tablet_manager.cc:1397] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:55:24.939795 18861 ts_tablet_manager.cc:1428] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 16:55:24.939805 18859 ts_tablet_manager.cc:1397] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 16:55:24.940435 18865 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:24.940546 18860 raft_consensus.cc:348] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:24.940619 18859 raft_consensus.cc:348] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:24.940793 18860 raft_consensus.cc:374] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:24.940871 18859 raft_consensus.cc:374] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:24.940883 18860 raft_consensus.cc:729] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Initialized, Role: FOLLOWER
I20260619 16:55:24.940992 18865 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:24.941144 18860 consensus_queue.cc:260] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:24.941313 18859 raft_consensus.cc:729] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708626cbf3d4bd29ad2078dcac27eeb, State: Initialized, Role: FOLLOWER
I20260619 16:55:24.941538 18847 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:24.941614 18859 consensus_queue.cc:260] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:24.941823 18847 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:24.942790 18860 ts_tablet_manager.cc:1428] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:24.944245 18859 ts_tablet_manager.cc:1428] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 16:55:24.944083 18866 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:24.944541 18866 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:25.059743 18847 raft_consensus.cc:482] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:25.059968 18847 raft_consensus.cc:504] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.060786 18847 leader_election.cc:288] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389), bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015)
I20260619 16:55:25.061403 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f157f6a93c44b8ab112f27c94feecf3" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" is_pre_election: true
I20260619 16:55:25.061407 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f157f6a93c44b8ab112f27c94feecf3" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b" is_pre_election: true
I20260619 16:55:25.061679 18635 raft_consensus.cc:2454] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:25.061717 18770 raft_consensus.cc:2454] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:25.062182 18421 leader_election.cc:302] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2708626cbf3d4bd29ad2078dcac27eeb, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:25.062484 18847 raft_consensus.cc:2781] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:25.062572 18847 raft_consensus.cc:482] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:25.062644 18847 raft_consensus.cc:3037] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:25.064661 18847 raft_consensus.cc:504] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.065263 18847 leader_election.cc:288] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Requested vote from peers 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389), bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015)
I20260619 16:55:25.065775 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f157f6a93c44b8ab112f27c94feecf3" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b"
I20260619 16:55:25.065776 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f157f6a93c44b8ab112f27c94feecf3" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb"
I20260619 16:55:25.065996 18770 raft_consensus.cc:3037] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:25.066017 18635 raft_consensus.cc:3037] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:25.068763 18770 raft_consensus.cc:2454] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:25.068773 18635 raft_consensus.cc:2454] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:25.069343 18423 leader_election.cc:302] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2708626cbf3d4bd29ad2078dcac27eeb, bd8485898450436b8467a7e5a723de7b, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:25.069744 18847 raft_consensus.cc:2781] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:25.069947 18847 raft_consensus.cc:686] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 LEADER]: Becoming Leader. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Running, Role: LEADER
I20260619 16:55:25.070219 18847 consensus_queue.cc:237] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.072399 18850 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: TxnStatusTablet state changed. Reason: New leader dbf05a8c520447da9f3aa3c9cdb1e924. Latest consensus state: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:25.072767 18850 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:55:25.073453 18868 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:55:25.073752 18868 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:55:25.074115 18334 catalog_manager.cc:5495] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1). New cstate: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } health_report { overall_health: UNKNOWN } } }
I20260619 16:55:25.095252 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:25.087917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.16.48.1:37037)
0619 16:55:25.088690 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:25.088695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:25.088714 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:55:25.088793 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:25.088797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:25.088880 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:25.089012 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:55:25.089464 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.090369 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.092409 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.093044 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.093163 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:25.093862 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:55:25.094536 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:55:25.094673 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:25.094879 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 16:55:25.094955 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":488,"threads_started":1}
I20260619 16:55:25.095307 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:25.087916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:37037 (local address 127.0.0.1:46868)
0619 16:55:25.088206 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:25.088228 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:25.088388 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:25.089080 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:25.089087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:25.089105 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:55:25.089300 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:25.089311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.090515 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.090523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:25.092209 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:25.092227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.093233 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.093240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:25.093479 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:25.093561 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:55:25.094668 (+ 1107us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:55:25.094695 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:55:25.094796 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:55:25.094966 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 16:55:25.105156 18635 raft_consensus.cc:1261] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:25.105156 18770 raft_consensus.cc:1261] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:25.106442 18850 consensus_queue.cc:1035] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:25.106899 18847 consensus_queue.cc:1035] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:25.130388 18866 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: TxnStatusTablet state changed. Reason: New leader dbf05a8c520447da9f3aa3c9cdb1e924. Latest consensus state: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:25.130697 18866 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:25.133553 18872 mvcc.cc:205] Tried to move back new op lower bound from 7298613760419020800 to 7298613760290308096. Current Snapshot: MvccSnapshot[applied={T|T < 7298613760419020800}]
I20260619 16:55:25.136279 18847 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:25.136648 18847 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:55:25.139328 18874 mvcc.cc:205] Tried to move back new op lower bound from 7298613760419020800 to 7298613760290308096. Current Snapshot: MvccSnapshot[applied={T|T < 7298613760419020800}]
I20260619 16:55:25.140992 18865 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: TxnStatusTablet state changed. Reason: New leader dbf05a8c520447da9f3aa3c9cdb1e924. Latest consensus state: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:25.141265 18865 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:25.143056 18850 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:25.143354 18850 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P dbf05a8c520447da9f3aa3c9cdb1e924: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:55:25.149952 18866 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:25.150255 18866 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P 2708626cbf3d4bd29ad2078dcac27eeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:25.153964 18865 tablet_replica.cc:439] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } } }
I20260619 16:55:25.154332 18865 tablet_replica.cc:442] T 9f157f6a93c44b8ab112f27c94feecf3 P bd8485898450436b8467a7e5a723de7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:55:25.161924 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:25.153197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.16.48.1:37037)
0619 16:55:25.153607 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:25.153614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:25.156365 (+ 2751us) server_negotiation.cc:407] Connection header received
0619 16:55:25.156737 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:25.156745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:25.156840 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:25.156905 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:55:25.157668 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.158500 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.160007 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.160561 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.160728 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:25.161178 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:55:25.161289 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:55:25.161404 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:25.161552 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:55:25.161639 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260619 16:55:25.183836 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:25.176258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:58579 (local address 127.16.48.1:37037)
0619 16:55:25.176480 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:25.176487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:25.177238 (+ 751us) server_negotiation.cc:407] Connection header received
0619 16:55:25.177519 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:25.177525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:25.177646 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:25.177767 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:55:25.178386 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.179600 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.181409 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.182203 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.182366 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:25.182946 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:55:25.183092 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:55:25.183205 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:25.183430 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 16:55:25.183507 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:55:25.184006 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:25.176238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:37037 (local address 127.16.48.1:58579)
0619 16:55:25.177110 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:25.177127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:25.177340 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:25.177782 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:25.177785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:25.177794 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:55:25.178138 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:25.178148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.179781 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.179785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:25.181234 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:25.181251 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:25.182451 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:25.182456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:25.182612 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:25.182707 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:55:25.183248 (+ 541us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:55:25.183258 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:55:25.183348 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:55:25.183509 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":476,"threads_started":1}
I20260619 16:55:25.196472 18770 raft_consensus.cc:1261] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:25.196677 18634 raft_consensus.cc:1261] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:25.197212 18847 consensus_queue.cc:1035] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:25.197611 18850 consensus_queue.cc:1035] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:25.229797 18873 mvcc.cc:205] Tried to move back new op lower bound from 7298613760798044160 to 7298613759038095360. Current Snapshot: MvccSnapshot[applied={T|T < 7298613760798044160}]
I20260619 16:55:25.288239 18334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:25.288493 18334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:25.292539 18334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a: Sending DeleteTablet for 3 replicas of tablet df4b6743774f462ebe3a472343de15bd
I20260619 16:55:25.293974 18750 tablet_service.cc:1514] Processing DeleteTablet for tablet df4b6743774f462ebe3a472343de15bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:25 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:51758
I20260619 16:55:25.294175 18470 tablet_service.cc:1514] Processing DeleteTablet for tablet df4b6743774f462ebe3a472343de15bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:25 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:46856
I20260619 16:55:25.294750 18610 tablet_service.cc:1514] Processing DeleteTablet for tablet df4b6743774f462ebe3a472343de15bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:25 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:60052
I20260619 16:55:25.295588 18334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:55:25.296634 18334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:25.298709 18890 tablet_replica.cc:330] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: stopping tablet replica
I20260619 16:55:25.298872 18889 tablet_replica.cc:330] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: stopping tablet replica
I20260619 16:55:25.299229 18890 raft_consensus.cc:2229] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:55:25.299278 18889 raft_consensus.cc:2229] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:25.299751 18889 raft_consensus.cc:2258] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:25.299945 18890 raft_consensus.cc:2258] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:25.300652 18891 tablet_replica.cc:330] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: stopping tablet replica
I20260619 16:55:25.301074 18891 raft_consensus.cc:2229] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:25.301420 18891 raft_consensus.cc:2258] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:25.302214 18889 ts_tablet_manager.cc:1905] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:25.303395 18890 ts_tablet_manager.cc:1905] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:25.307749 18470 tablet_service.cc:1467] Processing CreateTablet for tablet 86fa37daf94d45a0892745784c340e30 (DEFAULT_TABLE table=test-table [id=2acca18ab5ea4502a81a8aa404c39cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:25.308442 18470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86fa37daf94d45a0892745784c340e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:25.310510 18889 ts_tablet_manager.cc:1918] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:55:25.310830 18889 log.cc:1196] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/df4b6743774f462ebe3a472343de15bd
I20260619 16:55:25.311441 18889 ts_tablet_manager.cc:1939] T df4b6743774f462ebe3a472343de15bd P bd8485898450436b8467a7e5a723de7b: Deleting consensus metadata
I20260619 16:55:25.311756 18890 ts_tablet_manager.cc:1918] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:55:25.311942 18890 log.cc:1196] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/df4b6743774f462ebe3a472343de15bd
I20260619 16:55:25.312642 18610 tablet_service.cc:1467] Processing CreateTablet for tablet 86fa37daf94d45a0892745784c340e30 (DEFAULT_TABLE table=test-table [id=2acca18ab5ea4502a81a8aa404c39cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:25.313171 18610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86fa37daf94d45a0892745784c340e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:25.313712 18891 ts_tablet_manager.cc:1905] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:25.313949 18320 catalog_manager.cc:4841] TS bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015): tablet df4b6743774f462ebe3a472343de15bd (table test-table [id=7a99d33c10f64cc1a4794c662c5c7dc4]) successfully deleted
I20260619 16:55:25.314913 18890 ts_tablet_manager.cc:1939] T df4b6743774f462ebe3a472343de15bd P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting consensus metadata
I20260619 16:55:25.316870 18320 catalog_manager.cc:4841] TS dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1:37037): tablet df4b6743774f462ebe3a472343de15bd (table test-table [id=7a99d33c10f64cc1a4794c662c5c7dc4]) successfully deleted
I20260619 16:55:25.320935 18859 tablet_bootstrap.cc:492] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: Bootstrap starting.
I20260619 16:55:25.321223 18750 tablet_service.cc:1467] Processing CreateTablet for tablet 86fa37daf94d45a0892745784c340e30 (DEFAULT_TABLE table=test-table [id=2acca18ab5ea4502a81a8aa404c39cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:25.321568 18891 ts_tablet_manager.cc:1918] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:55:25.321699 18750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86fa37daf94d45a0892745784c340e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:25.321764 18891 log.cc:1196] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/df4b6743774f462ebe3a472343de15bd
I20260619 16:55:25.322359 18891 ts_tablet_manager.cc:1939] T df4b6743774f462ebe3a472343de15bd P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting consensus metadata
I20260619 16:55:25.322528 18859 tablet_bootstrap.cc:654] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:25.324455 18318 catalog_manager.cc:4841] TS 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389): tablet df4b6743774f462ebe3a472343de15bd (table test-table [id=7a99d33c10f64cc1a4794c662c5c7dc4]) successfully deleted
I20260619 16:55:25.325521 18860 tablet_bootstrap.cc:492] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: Bootstrap starting.
I20260619 16:55:25.325534 18859 tablet_bootstrap.cc:492] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: No bootstrap required, opened a new log
I20260619 16:55:25.325699 18859 ts_tablet_manager.cc:1397] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:55:25.326480 18859 raft_consensus.cc:348] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.326742 18859 raft_consensus.cc:374] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:25.326808 18859 raft_consensus.cc:729] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708626cbf3d4bd29ad2078dcac27eeb, State: Initialized, Role: FOLLOWER
I20260619 16:55:25.327616 18860 tablet_bootstrap.cc:654] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:25.328748 18861 tablet_bootstrap.cc:492] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: Bootstrap starting.
I20260619 16:55:25.328444 18859 consensus_queue.cc:260] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.329663 18859 ts_tablet_manager.cc:1428] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260619 16:55:25.330461 18861 tablet_bootstrap.cc:654] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:25.336750 18860 tablet_bootstrap.cc:492] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: No bootstrap required, opened a new log
I20260619 16:55:25.337371 18860 ts_tablet_manager.cc:1397] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260619 16:55:25.337601 18861 tablet_bootstrap.cc:492] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: No bootstrap required, opened a new log
I20260619 16:55:25.337805 18861 ts_tablet_manager.cc:1397] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 16:55:25.338228 18860 raft_consensus.cc:348] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.338497 18860 raft_consensus.cc:374] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:25.338598 18860 raft_consensus.cc:729] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Initialized, Role: FOLLOWER
I20260619 16:55:25.338662 18861 raft_consensus.cc:348] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.338972 18861 raft_consensus.cc:374] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:25.338877 18860 consensus_queue.cc:260] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.339066 18861 raft_consensus.cc:729] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8485898450436b8467a7e5a723de7b, State: Initialized, Role: FOLLOWER
I20260619 16:55:25.339318 18861 consensus_queue.cc:260] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.341600 18861 ts_tablet_manager.cc:1428] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:55:25.342118 18860 ts_tablet_manager.cc:1428] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:55:25.398558 18850 raft_consensus.cc:482] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:25.398804 18850 raft_consensus.cc:504] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.399524 18850 leader_election.cc:288] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389), bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015)
I20260619 16:55:25.400216 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86fa37daf94d45a0892745784c340e30" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b" is_pre_election: true
I20260619 16:55:25.400276 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86fa37daf94d45a0892745784c340e30" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" is_pre_election: true
I20260619 16:55:25.400506 18635 raft_consensus.cc:2454] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:25.400486 18768 raft_consensus.cc:2454] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:25.401014 18423 leader_election.cc:302] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8485898450436b8467a7e5a723de7b, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:25.401360 18850 raft_consensus.cc:2781] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:25.401537 18850 raft_consensus.cc:482] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:25.401657 18850 raft_consensus.cc:3037] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:25.405270 18850 raft_consensus.cc:504] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.406231 18850 leader_election.cc:288] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Requested vote from peers 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389), bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015)
I20260619 16:55:25.406787 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86fa37daf94d45a0892745784c340e30" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b"
I20260619 16:55:25.406801 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86fa37daf94d45a0892745784c340e30" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb"
I20260619 16:55:25.407017 18768 raft_consensus.cc:3037] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:25.407099 18635 raft_consensus.cc:3037] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:25.409468 18768 raft_consensus.cc:2454] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:25.410048 18423 leader_election.cc:302] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8485898450436b8467a7e5a723de7b, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:25.410372 18635 raft_consensus.cc:2454] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:25.410459 18850 raft_consensus.cc:2781] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:25.410656 18850 raft_consensus.cc:686] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 LEADER]: Becoming Leader. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Running, Role: LEADER
I20260619 16:55:25.411303 18850 consensus_queue.cc:237] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:25.414512 18333 catalog_manager.cc:5495] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1). New cstate: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } health_report { overall_health: UNKNOWN } } }
I20260619 16:55:25.448344 18768 raft_consensus.cc:1261] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:25.448801 18634 raft_consensus.cc:1261] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:25.449105 18847 consensus_queue.cc:1035] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:25.449710 18850 consensus_queue.cc:1035] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:55:25.482391 18872 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:55:27.077198 18333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:27.077433 18333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:27.081799 18333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a: Sending DeleteTablet for 3 replicas of tablet 86fa37daf94d45a0892745784c340e30
I20260619 16:55:27.082681 18470 tablet_service.cc:1514] Processing DeleteTablet for tablet 86fa37daf94d45a0892745784c340e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:27 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:46856
I20260619 16:55:27.082858 18750 tablet_service.cc:1514] Processing DeleteTablet for tablet 86fa37daf94d45a0892745784c340e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:27 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:51758
I20260619 16:55:27.082999 18610 tablet_service.cc:1514] Processing DeleteTablet for tablet 86fa37daf94d45a0892745784c340e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:27 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:60052
I20260619 16:55:27.083711 18920 tablet_replica.cc:330] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: stopping tablet replica
I20260619 16:55:27.083854 18919 tablet_replica.cc:330] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: stopping tablet replica
I20260619 16:55:27.084031 18919 raft_consensus.cc:2229] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:27.084017 18920 raft_consensus.cc:2229] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:27.084317 18919 raft_consensus.cc:2258] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:27.084354 18920 raft_consensus.cc:2258] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:27.084893 18918 tablet_replica.cc:330] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: stopping tablet replica
I20260619 16:55:27.085111 18918 raft_consensus.cc:2229] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:55:27.085521 18918 raft_consensus.cc:2258] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:27.085562 18919 ts_tablet_manager.cc:1905] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:27.086536 18918 ts_tablet_manager.cc:1905] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:27.092271 18920 ts_tablet_manager.cc:1905] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:27.093178 18918 ts_tablet_manager.cc:1918] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:55:27.093321 18918 log.cc:1196] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/86fa37daf94d45a0892745784c340e30
I20260619 16:55:27.093753 18918 ts_tablet_manager.cc:1939] T 86fa37daf94d45a0892745784c340e30 P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting consensus metadata
I20260619 16:55:27.095090 18320 catalog_manager.cc:4841] TS dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1:37037): tablet 86fa37daf94d45a0892745784c340e30 (table test-table [id=2acca18ab5ea4502a81a8aa404c39cef]) successfully deleted
I20260619 16:55:27.096377 18919 ts_tablet_manager.cc:1918] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:55:27.096530 18919 log.cc:1196] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/86fa37daf94d45a0892745784c340e30
I20260619 16:55:27.097052 18919 ts_tablet_manager.cc:1939] T 86fa37daf94d45a0892745784c340e30 P bd8485898450436b8467a7e5a723de7b: Deleting consensus metadata
I20260619 16:55:27.098335 18320 catalog_manager.cc:4841] TS bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015): tablet 86fa37daf94d45a0892745784c340e30 (table test-table [id=2acca18ab5ea4502a81a8aa404c39cef]) successfully deleted
I20260619 16:55:27.099282 18920 ts_tablet_manager.cc:1918] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:55:27.099412 18920 log.cc:1196] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/86fa37daf94d45a0892745784c340e30
I20260619 16:55:27.099823 18920 ts_tablet_manager.cc:1939] T 86fa37daf94d45a0892745784c340e30 P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting consensus metadata
I20260619 16:55:27.101106 18318 catalog_manager.cc:4841] TS 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389): tablet 86fa37daf94d45a0892745784c340e30 (table test-table [id=2acca18ab5ea4502a81a8aa404c39cef]) successfully deleted
Jun 19 16:55:27 dist-test-slave-3ffd krb5kdc[18289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:55:27 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:27.138149 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:27.121986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.16.48.62:40191)
0619 16:55:27.122859 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:27.122864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:27.122884 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:55:27.122936 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:27.122940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:27.122997 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:27.123118 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:27.124684 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:27.125558 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:27.126841 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:27.127677 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:27.127836 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:27.132307 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:27.132315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:27.132320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:27.132423 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:27.135162 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:27.136073 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:27.136080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:27.136084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:27.136301 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:27.136801 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:27.136805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:27.136808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:27.137134 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:27.137345 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:27.137675 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 16:55:27.137872 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":558,"threads_started":1}
I20260619 16:55:27.142216 18333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:55:27.142918 18333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:27.150115 18610 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb6c53a86da476d9a3883642ee2744d (DEFAULT_TABLE table=test-table [id=afebd2ed11c74bc99c5a3b4c8773cd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:27.150115 18470 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb6c53a86da476d9a3883642ee2744d (DEFAULT_TABLE table=test-table [id=afebd2ed11c74bc99c5a3b4c8773cd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:27.150388 18750 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb6c53a86da476d9a3883642ee2744d (DEFAULT_TABLE table=test-table [id=afebd2ed11c74bc99c5a3b4c8773cd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:27.150583 18470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb6c53a86da476d9a3883642ee2744d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:27.150583 18610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb6c53a86da476d9a3883642ee2744d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:27.150828 18750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb6c53a86da476d9a3883642ee2744d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:27.158485 18931 tablet_bootstrap.cc:492] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: Bootstrap starting.
I20260619 16:55:27.159276 18930 tablet_bootstrap.cc:492] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: Bootstrap starting.
I20260619 16:55:27.159771 18929 tablet_bootstrap.cc:492] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: Bootstrap starting.
I20260619 16:55:27.161445 18931 tablet_bootstrap.cc:654] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:27.161445 18930 tablet_bootstrap.cc:654] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:27.162101 18929 tablet_bootstrap.cc:654] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:27.163995 18930 tablet_bootstrap.cc:492] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: No bootstrap required, opened a new log
I20260619 16:55:27.164209 18930 ts_tablet_manager.cc:1397] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260619 16:55:27.164413 18929 tablet_bootstrap.cc:492] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: No bootstrap required, opened a new log
I20260619 16:55:27.164593 18929 ts_tablet_manager.cc:1397] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260619 16:55:27.165107 18930 raft_consensus.cc:348] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.165333 18931 tablet_bootstrap.cc:492] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: No bootstrap required, opened a new log
I20260619 16:55:27.165419 18930 raft_consensus.cc:374] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:27.165247 18929 raft_consensus.cc:348] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.165493 18931 ts_tablet_manager.cc:1397] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:55:27.165513 18930 raft_consensus.cc:729] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708626cbf3d4bd29ad2078dcac27eeb, State: Initialized, Role: FOLLOWER
I20260619 16:55:27.165512 18929 raft_consensus.cc:374] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:27.165606 18929 raft_consensus.cc:729] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8485898450436b8467a7e5a723de7b, State: Initialized, Role: FOLLOWER
I20260619 16:55:27.165766 18930 consensus_queue.cc:260] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.165864 18929 consensus_queue.cc:260] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.166406 18931 raft_consensus.cc:348] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.166744 18931 raft_consensus.cc:374] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:27.166857 18931 raft_consensus.cc:729] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Initialized, Role: FOLLOWER
I20260619 16:55:27.167157 18931 consensus_queue.cc:260] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.168008 18931 ts_tablet_manager.cc:1428] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:55:27.169580 18929 ts_tablet_manager.cc:1428] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260619 16:55:27.170418 18930 ts_tablet_manager.cc:1428] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 16:55:27.206811 18901 raft_consensus.cc:482] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:27.207085 18901 raft_consensus.cc:504] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.207849 18901 leader_election.cc:288] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389), bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015)
I20260619 16:55:27.208487 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb6c53a86da476d9a3883642ee2744d" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b" is_pre_election: true
I20260619 16:55:27.208753 18769 raft_consensus.cc:2454] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:27.208892 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb6c53a86da476d9a3883642ee2744d" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" is_pre_election: true
I20260619 16:55:27.209185 18634 raft_consensus.cc:2454] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:27.209271 18423 leader_election.cc:302] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8485898450436b8467a7e5a723de7b, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:27.209611 18901 raft_consensus.cc:2781] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:27.209789 18901 raft_consensus.cc:482] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:27.209908 18901 raft_consensus.cc:3037] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:27.212345 18901 raft_consensus.cc:504] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.213238 18901 leader_election.cc:288] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Requested vote from peers 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389), bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015)
I20260619 16:55:27.213829 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb6c53a86da476d9a3883642ee2744d" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb"
I20260619 16:55:27.213907 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb6c53a86da476d9a3883642ee2744d" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b"
I20260619 16:55:27.214126 18634 raft_consensus.cc:3037] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:27.214143 18769 raft_consensus.cc:3037] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:27.217808 18634 raft_consensus.cc:2454] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:27.217815 18769 raft_consensus.cc:2454] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:27.218502 18423 leader_election.cc:302] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8485898450436b8467a7e5a723de7b, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:27.219050 18901 raft_consensus.cc:2781] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:27.219216 18901 raft_consensus.cc:686] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 LEADER]: Becoming Leader. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Running, Role: LEADER
I20260619 16:55:27.219496 18901 consensus_queue.cc:237] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.222863 18334 catalog_manager.cc:5495] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1). New cstate: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } health_report { overall_health: UNKNOWN } } }
I20260619 16:55:27.256476 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:27.251092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.16.48.1:37037)
0619 16:55:27.251778 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:27.251783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:27.251808 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:55:27.251885 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:27.251890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:27.251972 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:27.252080 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:55:27.252509 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:27.253283 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:27.254600 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:27.255051 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:27.255200 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:27.255666 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:55:27.255769 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:55:27.255884 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:27.256042 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 16:55:27.256157 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":431,"threads_started":1}
I20260619 16:55:27.266021 18634 raft_consensus.cc:1261] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:27.266036 18769 raft_consensus.cc:1261] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:27.266613 18847 consensus_queue.cc:1035] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:27.266906 18907 consensus_queue.cc:1035] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:27.315778 18333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:27.316009 18333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:27.319082 18333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a: Sending DeleteTablet for 3 replicas of tablet 4cb6c53a86da476d9a3883642ee2744d
I20260619 16:55:27.320062 18610 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb6c53a86da476d9a3883642ee2744d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:27 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:60052
I20260619 16:55:27.320098 18470 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb6c53a86da476d9a3883642ee2744d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:27 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:46856
I20260619 16:55:27.320165 18750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb6c53a86da476d9a3883642ee2744d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:27 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:51758
I20260619 16:55:27.320403 18920 tablet_replica.cc:330] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: stopping tablet replica
I20260619 16:55:27.320612 18919 tablet_replica.cc:330] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: stopping tablet replica
I20260619 16:55:27.320595 18920 raft_consensus.cc:2229] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:27.320801 18920 raft_consensus.cc:2258] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:27.320825 18919 raft_consensus.cc:2229] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:27.320924 18918 tablet_replica.cc:330] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: stopping tablet replica
I20260619 16:55:27.321019 18919 raft_consensus.cc:2258] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:27.321126 18918 raft_consensus.cc:2229] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:55:27.321513 18918 raft_consensus.cc:2258] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:27.321733 18333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:55:27.322264 18920 ts_tablet_manager.cc:1905] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:55:27.322500 18333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:27.322607 18918 ts_tablet_manager.cc:1905] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:27.323022 18919 ts_tablet_manager.cc:1905] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:27.329203 18920 ts_tablet_manager.cc:1918] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:55:27.329399 18920 log.cc:1196] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/4cb6c53a86da476d9a3883642ee2744d
I20260619 16:55:27.329865 18920 ts_tablet_manager.cc:1939] T 4cb6c53a86da476d9a3883642ee2744d P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting consensus metadata
I20260619 16:55:27.330230 18918 ts_tablet_manager.cc:1918] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:55:27.330389 18918 log.cc:1196] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/4cb6c53a86da476d9a3883642ee2744d
I20260619 16:55:27.330847 18918 ts_tablet_manager.cc:1939] T 4cb6c53a86da476d9a3883642ee2744d P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting consensus metadata
I20260619 16:55:27.331153 18318 catalog_manager.cc:4841] TS 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389): tablet 4cb6c53a86da476d9a3883642ee2744d (table test-table [id=afebd2ed11c74bc99c5a3b4c8773cd6a]) successfully deleted
I20260619 16:55:27.331640 18610 tablet_service.cc:1467] Processing CreateTablet for tablet 392155b596f747f3afd11cd5e002ec19 (DEFAULT_TABLE table=test-table [id=1b7857ec5f7f4a3d86bda5256735d7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:27.332058 18320 catalog_manager.cc:4841] TS dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1:37037): tablet 4cb6c53a86da476d9a3883642ee2744d (table test-table [id=afebd2ed11c74bc99c5a3b4c8773cd6a]) successfully deleted
I20260619 16:55:27.332150 18610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392155b596f747f3afd11cd5e002ec19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:27.332101 18750 tablet_service.cc:1467] Processing CreateTablet for tablet 392155b596f747f3afd11cd5e002ec19 (DEFAULT_TABLE table=test-table [id=1b7857ec5f7f4a3d86bda5256735d7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:27.332538 18750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392155b596f747f3afd11cd5e002ec19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:27.332965 18470 tablet_service.cc:1467] Processing CreateTablet for tablet 392155b596f747f3afd11cd5e002ec19 (DEFAULT_TABLE table=test-table [id=1b7857ec5f7f4a3d86bda5256735d7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:27.333767 18470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392155b596f747f3afd11cd5e002ec19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:27.333796 18919 ts_tablet_manager.cc:1918] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:55:27.333977 18919 log.cc:1196] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/4cb6c53a86da476d9a3883642ee2744d
I20260619 16:55:27.334430 18919 ts_tablet_manager.cc:1939] T 4cb6c53a86da476d9a3883642ee2744d P bd8485898450436b8467a7e5a723de7b: Deleting consensus metadata
I20260619 16:55:27.336890 18320 catalog_manager.cc:4841] TS bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015): tablet 4cb6c53a86da476d9a3883642ee2744d (table test-table [id=afebd2ed11c74bc99c5a3b4c8773cd6a]) successfully deleted
I20260619 16:55:27.338358 18930 tablet_bootstrap.cc:492] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: Bootstrap starting.
I20260619 16:55:27.340018 18930 tablet_bootstrap.cc:654] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:27.340299 18929 tablet_bootstrap.cc:492] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: Bootstrap starting.
I20260619 16:55:27.340363 18931 tablet_bootstrap.cc:492] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: Bootstrap starting.
I20260619 16:55:27.342299 18931 tablet_bootstrap.cc:654] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:27.342491 18929 tablet_bootstrap.cc:654] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:27.342957 18930 tablet_bootstrap.cc:492] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: No bootstrap required, opened a new log
I20260619 16:55:27.343140 18930 ts_tablet_manager.cc:1397] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:55:27.343941 18930 raft_consensus.cc:348] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.344256 18930 raft_consensus.cc:374] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:27.344355 18930 raft_consensus.cc:729] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708626cbf3d4bd29ad2078dcac27eeb, State: Initialized, Role: FOLLOWER
I20260619 16:55:27.344709 18931 tablet_bootstrap.cc:492] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: No bootstrap required, opened a new log
I20260619 16:55:27.344648 18930 consensus_queue.cc:260] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.344887 18931 ts_tablet_manager.cc:1397] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:55:27.345263 18930 ts_tablet_manager.cc:1428] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 16:55:27.345649 18929 tablet_bootstrap.cc:492] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: No bootstrap required, opened a new log
I20260619 16:55:27.345845 18929 ts_tablet_manager.cc:1397] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 16:55:27.345721 18931 raft_consensus.cc:348] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.345969 18931 raft_consensus.cc:374] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:27.346071 18931 raft_consensus.cc:729] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Initialized, Role: FOLLOWER
I20260619 16:55:27.346345 18931 consensus_queue.cc:260] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.346590 18929 raft_consensus.cc:348] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.346786 18931 ts_tablet_manager.cc:1428] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:55:27.346863 18929 raft_consensus.cc:374] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:27.346943 18929 raft_consensus.cc:729] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8485898450436b8467a7e5a723de7b, State: Initialized, Role: FOLLOWER
I20260619 16:55:27.347190 18929 consensus_queue.cc:260] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.349009 18929 ts_tablet_manager.cc:1428] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 16:55:27.584214 18901 raft_consensus.cc:482] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:27.584465 18901 raft_consensus.cc:504] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.585160 18901 leader_election.cc:288] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389), bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015)
I20260619 16:55:27.585924 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392155b596f747f3afd11cd5e002ec19" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" is_pre_election: true
I20260619 16:55:27.585924 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392155b596f747f3afd11cd5e002ec19" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b" is_pre_election: true
I20260619 16:55:27.586218 18634 raft_consensus.cc:2454] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:27.586218 18769 raft_consensus.cc:2454] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 0.
I20260619 16:55:27.586766 18423 leader_election.cc:302] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2708626cbf3d4bd29ad2078dcac27eeb, bd8485898450436b8467a7e5a723de7b, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:27.587149 18901 raft_consensus.cc:2781] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:27.587240 18901 raft_consensus.cc:482] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:27.587384 18901 raft_consensus.cc:3037] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:27.589494 18901 raft_consensus.cc:504] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.590102 18901 leader_election.cc:288] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Requested vote from peers 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389), bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015)
I20260619 16:55:27.590786 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392155b596f747f3afd11cd5e002ec19" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708626cbf3d4bd29ad2078dcac27eeb"
I20260619 16:55:27.590786 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392155b596f747f3afd11cd5e002ec19" candidate_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8485898450436b8467a7e5a723de7b"
I20260619 16:55:27.591050 18634 raft_consensus.cc:3037] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:27.591070 18769 raft_consensus.cc:3037] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:27.594513 18769 raft_consensus.cc:2454] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:27.594513 18634 raft_consensus.cc:2454] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf05a8c520447da9f3aa3c9cdb1e924 in term 1.
I20260619 16:55:27.595096 18421 leader_election.cc:302] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2708626cbf3d4bd29ad2078dcac27eeb, dbf05a8c520447da9f3aa3c9cdb1e924; no voters:
I20260619 16:55:27.595492 18901 raft_consensus.cc:2781] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:27.595698 18901 raft_consensus.cc:686] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 LEADER]: Becoming Leader. State: Replica: dbf05a8c520447da9f3aa3c9cdb1e924, State: Running, Role: LEADER
I20260619 16:55:27.595970 18901 consensus_queue.cc:237] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } }
I20260619 16:55:27.598882 18334 catalog_manager.cc:5495] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1). New cstate: current_term: 1 leader_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf05a8c520447da9f3aa3c9cdb1e924" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 } health_report { overall_health: UNKNOWN } } }
I20260619 16:55:27.655654 18634 raft_consensus.cc:1261] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:27.655682 18768 raft_consensus.cc:1261] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Refusing update from remote peer dbf05a8c520447da9f3aa3c9cdb1e924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:27.656441 18847 consensus_queue.cc:1035] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2708626cbf3d4bd29ad2078dcac27eeb" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 39389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:27.656739 18901 consensus_queue.cc:1035] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8485898450436b8467a7e5a723de7b" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:55:27.689087 18899 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:55:29.286477 18334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:29.286748 18334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:29.290796 18334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be52940b5bab450b8941f76b2dacf01a: Sending DeleteTablet for 3 replicas of tablet 392155b596f747f3afd11cd5e002ec19
I20260619 16:55:29.291726 18470 tablet_service.cc:1514] Processing DeleteTablet for tablet 392155b596f747f3afd11cd5e002ec19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:29 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:46856
I20260619 16:55:29.291972 18610 tablet_service.cc:1514] Processing DeleteTablet for tablet 392155b596f747f3afd11cd5e002ec19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:29 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:60052
I20260619 16:55:29.292023 18750 tablet_service.cc:1514] Processing DeleteTablet for tablet 392155b596f747f3afd11cd5e002ec19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:29 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:51758
I20260619 16:55:29.292533 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18405
I20260619 16:55:29.309018 18972 tablet_replica.cc:330] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: stopping tablet replica
I20260619 16:55:29.309267 18971 tablet_replica.cc:330] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: stopping tablet replica
I20260619 16:55:29.309307 18972 raft_consensus.cc:2229] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:29.309485 18971 raft_consensus.cc:2229] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:29.309614 18972 raft_consensus.cc:2258] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:29.309818 18971 raft_consensus.cc:2258] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:29.311127 18972 ts_tablet_manager.cc:1905] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:29.311180 18971 ts_tablet_manager.cc:1905] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:29.312346 18970 tablet_replica.cc:330] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: stopping tablet replica
I20260619 16:55:29.312608 18970 raft_consensus.cc:2229] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:55:29.313030 18970 raft_consensus.cc:2258] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:29.314038 18970 ts_tablet_manager.cc:1905] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:29.317312 18972 ts_tablet_manager.cc:1918] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:55:29.317507 18972 log.cc:1196] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/392155b596f747f3afd11cd5e002ec19
I20260619 16:55:29.317891 18971 ts_tablet_manager.cc:1918] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:55:29.317947 18972 ts_tablet_manager.cc:1939] T 392155b596f747f3afd11cd5e002ec19 P bd8485898450436b8467a7e5a723de7b: Deleting consensus metadata
I20260619 16:55:29.318049 18971 log.cc:1196] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/392155b596f747f3afd11cd5e002ec19
I20260619 16:55:29.318473 18971 ts_tablet_manager.cc:1939] T 392155b596f747f3afd11cd5e002ec19 P 2708626cbf3d4bd29ad2078dcac27eeb: Deleting consensus metadata
I20260619 16:55:29.319150 18320 catalog_manager.cc:4841] TS bd8485898450436b8467a7e5a723de7b (127.16.48.3:39015): tablet 392155b596f747f3afd11cd5e002ec19 (table test-table [id=1b7857ec5f7f4a3d86bda5256735d7ad]) successfully deleted
I20260619 16:55:29.319536 18318 catalog_manager.cc:4841] TS 2708626cbf3d4bd29ad2078dcac27eeb (127.16.48.2:39389): tablet 392155b596f747f3afd11cd5e002ec19 (table test-table [id=1b7857ec5f7f4a3d86bda5256735d7ad]) successfully deleted
I20260619 16:55:29.321241 18970 ts_tablet_manager.cc:1918] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:55:29.321401 18970 log.cc:1196] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/392155b596f747f3afd11cd5e002ec19
I20260619 16:55:29.321794 18970 ts_tablet_manager.cc:1939] T 392155b596f747f3afd11cd5e002ec19 P dbf05a8c520447da9f3aa3c9cdb1e924: Deleting consensus metadata
I20260619 16:55:29.322767 18320 catalog_manager.cc:4841] TS dbf05a8c520447da9f3aa3c9cdb1e924 (127.16.48.1:37037): tablet 392155b596f747f3afd11cd5e002ec19 (table test-table [id=1b7857ec5f7f4a3d86bda5256735d7ad]) successfully deleted
Jun 19 16:55:29 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:29.330606 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:29.295127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.16.48.1:37037)
0619 16:55:29.307472 (+ 12345us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:29.307482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:29.307522 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:55:29.307602 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:29.307607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:29.308113 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:29.308331 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:29.310181 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:29.311387 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:29.313678 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:29.314364 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:29.314523 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:29.324524 (+ 10001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:29.324541 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:29.324557 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:29.324601 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:29.328087 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:29.328856 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:29.328864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:29.328866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:29.329006 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:29.329344 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:29.329347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:29.329349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:29.329700 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:29.329915 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:29.330167 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 16:55:29.330339 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12124,"thread_start_us":13324,"threads_started":1}
W20260619 16:55:29.331290 18536 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45796
W20260619 16:55:29.332039 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18405: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:55:29.332317 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 18405
I20260619 16:55:29.358543 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18545
Jun 19 16:55:29 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:55:29.375912 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:29.359378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.16.48.2:39389)
0619 16:55:29.359916 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:29.359921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:29.359936 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:55:29.359988 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:29.359991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:29.360054 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:29.360220 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:29.361463 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:29.362252 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:29.363655 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:29.364063 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:29.364268 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:29.369319 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:29.369338 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:29.369355 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:29.369400 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:29.372682 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:29.373517 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:29.373526 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:29.373529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:29.373756 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:29.374194 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:29.374200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:29.374211 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:29.374573 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:29.374917 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:29.375268 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 16:55:29.375552 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":279,"threads_started":1}
W20260619 16:55:29.376752 18675 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40088
W20260619 16:55:29.377470 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:55:29.377641 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 18545
I20260619 16:55:29.396102 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18684
Jun 19 16:55:29 dist-test-slave-3ffd krb5kdc[18289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:55:29.412329 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:29.397022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.16.48.3:39015)
0619 16:55:29.397630 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:29.397635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:29.397653 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:55:29.397708 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:29.397712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:29.397772 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:29.397866 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:29.399104 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:29.399866 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:29.401274 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:29.401753 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:29.401874 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:29.406339 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:29.406356 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:29.406373 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:29.406420 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:29.409293 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:29.410088 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:29.410097 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:29.410099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:29.410251 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:29.410659 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:29.410664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:29.410670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:29.411060 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:29.411364 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:29.411704 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 16:55:29.411935 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":331,"threads_started":1}
W20260619 16:55:29.413053 18815 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46556
W20260619 16:55:29.413688 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18684: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:55:29.413849 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 18684
I20260619 16:55:29.431903 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18302
I20260619 16:55:29.433332 18374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:29.548839 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 18302
2026-06-19T16:55:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8870 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:29 dist-test-slave-3ffd krb5kdc[18980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:29 dist-test-slave-3ffd krb5kdc[18980](info): set up 2 sockets
Jun 19 16:55:29 dist-test-slave-3ffd krb5kdc[18980](info): commencing operation
krb5kdc: starting...
W20260619 16:55:30.640810 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.006s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:30 dist-test-slave-3ffd krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:30.847409 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46597
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:46597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:31.193435 18993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:31.193820 18993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:31.193996 18993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:31.201721 18993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:31.201915 18993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:31.202034 18993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:31.202145 18993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:31.202229 18993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:31.212895 18993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:46597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:31.214372 18993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:31.215852 18993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:31.228801 18999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:31.228971 19001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:31.229007 18998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:31.230086 18993 server_base.cc:1034] running on GCE node
I20260619 16:55:31.230932 18993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:31.232589 18993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:31.233843 18993 hybrid_clock.cc:648] HybridClock initialized: now 1781888131233784 us; error 44 us; skew 500 ppm
I20260619 16:55:31.234272 18993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:31 dist-test-slave-3ffd krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888131, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:31.237739 18993 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:31.239531 18993 webserver.cc:460] Webserver started at http://127.16.48.62:36031/ using document root <none> and password file <none>
I20260619 16:55:31.240394 18993 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:31.240513 18993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:31.240805 18993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:31.243966 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "666a15994bee4b4696e666c7e3fd5530"
format_stamp: "Formatted at 2026-06-19 16:55:31 on dist-test-slave-3ffd"
server_key: "a8f89bf12bf76a43eea5af17f90f31c6"
server_key_iv: "60dd276f3c9a38f72b4d8f0a125bb18b"
server_key_version: "encryptionkey@0"
I20260619 16:55:31.244887 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "666a15994bee4b4696e666c7e3fd5530"
format_stamp: "Formatted at 2026-06-19 16:55:31 on dist-test-slave-3ffd"
server_key: "a8f89bf12bf76a43eea5af17f90f31c6"
server_key_iv: "60dd276f3c9a38f72b4d8f0a125bb18b"
server_key_version: "encryptionkey@0"
I20260619 16:55:31.250761 18993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 16:55:31.254742 19008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:31.256433 18993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:31.256647 18993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "666a15994bee4b4696e666c7e3fd5530"
format_stamp: "Formatted at 2026-06-19 16:55:31 on dist-test-slave-3ffd"
server_key: "a8f89bf12bf76a43eea5af17f90f31c6"
server_key_iv: "60dd276f3c9a38f72b4d8f0a125bb18b"
server_key_version: "encryptionkey@0"
I20260619 16:55:31.256908 18993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:31.273694 18993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:31.278800 18993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:31.279145 18993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:31.298554 18993 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:46597
I20260619 16:55:31.298576 19059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:46597 every 8 connection(s)
I20260619 16:55:31.300172 18993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:31.304481 19060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:31.307663 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 18993
I20260619 16:55:31.307924 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:31.308703 16576 external_mini_cluster.cc:1428] Setting key 82d2b1db01dd4069c48f853dd3251bec
I20260619 16:55:31.318143 19060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530: Bootstrap starting.
I20260619 16:55:31.322407 19060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:31.323942 19060 log.cc:824] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:55:31 dist-test-slave-3ffd krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:31.327381 19060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530: No bootstrap required, opened a new log
I20260619 16:55:31.334331 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:31.310939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.16.48.62:46597)
0619 16:55:31.311789 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:31.311803 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:31.311847 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:55:31.313681 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:31.313705 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:31.314171 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:31.314552 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:31.316447 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:31.317858 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:31.319694 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:31.320566 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:31.320731 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:31.325973 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:31.325981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:31.325999 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:31.326036 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:31.329397 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:31.330392 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:31.330402 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:31.330409 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:31.330616 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:31.331096 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:31.331101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:31.331103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:31.331793 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:31.332027 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:31.332585 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 16:55:31.333085 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":380,"threads_started":1}
I20260619 16:55:31.337363 19060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666a15994bee4b4696e666c7e3fd5530" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46597 } }
I20260619 16:55:31.337898 19060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:31.338068 19060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666a15994bee4b4696e666c7e3fd5530, State: Initialized, Role: FOLLOWER
I20260619 16:55:31.339021 19060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666a15994bee4b4696e666c7e3fd5530" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46597 } }
I20260619 16:55:31.339344 19060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:31.339511 19060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:31.339708 19060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:31.342072 19060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666a15994bee4b4696e666c7e3fd5530" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46597 } }
I20260619 16:55:31.342713 19060 leader_election.cc:302] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 666a15994bee4b4696e666c7e3fd5530; no voters:
I20260619 16:55:31.343367 19060 leader_election.cc:288] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:31.343587 19065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:31.345072 19065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [term 1 LEADER]: Becoming Leader. State: Replica: 666a15994bee4b4696e666c7e3fd5530, State: Running, Role: LEADER
I20260619 16:55:31.345676 19065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666a15994bee4b4696e666c7e3fd5530" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46597 } }
I20260619 16:55:31.346031 19060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:31.349174 19066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "666a15994bee4b4696e666c7e3fd5530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666a15994bee4b4696e666c7e3fd5530" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46597 } } }
I20260619 16:55:31.349156 19067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 666a15994bee4b4696e666c7e3fd5530. Latest consensus state: current_term: 1 leader_uuid: "666a15994bee4b4696e666c7e3fd5530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666a15994bee4b4696e666c7e3fd5530" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46597 } } }
I20260619 16:55:31.349699 19066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:31.349712 19067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:31.352633 19074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:31.359611 19074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:31.370975 19074 catalog_manager.cc:1347] Generated new cluster ID: 939e023a4bcc4589889047afdb370714
I20260619 16:55:31.371130 19074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:31.389096 19074 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:31.390640 19074 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:31.397840 19074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 666a15994bee4b4696e666c7e3fd5530: Generated new TSK 0
I20260619 16:55:31.398809 19074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:55:31.461937 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46597
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:31.814531 19086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:31.814862 19086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:31.814957 19086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:31.822355 19086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:31.822491 19086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:31.822661 19086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:31.833242 19086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:31.834722 19086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:31.836290 19086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:31.852452 19091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:31.852391 19094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:31.852473 19092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:31.853639 19086 server_base.cc:1034] running on GCE node
I20260619 16:55:31.854336 19086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:31.855410 19086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:31.856683 19086 hybrid_clock.cc:648] HybridClock initialized: now 1781888131856622 us; error 40 us; skew 500 ppm
I20260619 16:55:31.857079 19086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:31 dist-test-slave-3ffd krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888131, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:31.860548 19086 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:55:31.862140 19086 webserver.cc:460] Webserver started at http://127.16.48.1:35987/ using document root <none> and password file <none>
I20260619 16:55:31.862908 19086 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:31.863060 19086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:31.863390 19086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:31.866595 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "41922d6d9638416cabc2595df50ea0a5"
format_stamp: "Formatted at 2026-06-19 16:55:31 on dist-test-slave-3ffd"
server_key: "9d9a88ca3dd4be29ac1cc99476ccdb52"
server_key_iv: "9978d105dde071166c685538fcca0b63"
server_key_version: "encryptionkey@0"
I20260619 16:55:31.867484 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "41922d6d9638416cabc2595df50ea0a5"
format_stamp: "Formatted at 2026-06-19 16:55:31 on dist-test-slave-3ffd"
server_key: "9d9a88ca3dd4be29ac1cc99476ccdb52"
server_key_iv: "9978d105dde071166c685538fcca0b63"
server_key_version: "encryptionkey@0"
I20260619 16:55:31.873310 19086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 16:55:31.877259 19101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:31.878885 19086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:55:31.879110 19086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "41922d6d9638416cabc2595df50ea0a5"
format_stamp: "Formatted at 2026-06-19 16:55:31 on dist-test-slave-3ffd"
server_key: "9d9a88ca3dd4be29ac1cc99476ccdb52"
server_key_iv: "9978d105dde071166c685538fcca0b63"
server_key_version: "encryptionkey@0"
I20260619 16:55:31.879347 19086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:31.897493 19086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:31.902331 19086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:31.902698 19086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:31.903787 19086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:31.905346 19086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:31.905460 19086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:31.905613 19086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:31.905691 19086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:31.941517 19086 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:38299
I20260619 16:55:31.941562 19213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:38299 every 8 connection(s)
I20260619 16:55:31.942975 19086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:31.945236 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 19086
I20260619 16:55:31.945487 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:31.946295 16576 external_mini_cluster.cc:1428] Setting key b7b0a2e017fe94038636e3be5ce6f178
Jun 19 16:55:31 dist-test-slave-3ffd krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888131, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 16:55:31.967461 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:31.946162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:60413 (local address 127.16.48.62:46597)
0619 16:55:31.946699 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:31.946707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:31.948178 (+ 1471us) server_negotiation.cc:407] Connection header received
0619 16:55:31.950154 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:31.950162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:31.950265 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:31.950395 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:31.952094 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:31.953421 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:31.955784 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:31.956655 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:31.956824 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:31.960937 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:31.960947 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:31.960954 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:31.961004 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:31.963995 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:31.964864 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:31.964876 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:31.964880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:31.965002 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:31.965492 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:31.965496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:31.965498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:31.965839 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:31.965994 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:31.966979 (+ 985us) server_negotiation.cc:299] Negotiation successful
0619 16:55:31.967179 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":252,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:55:31.968966 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:31.946928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46597 (local address 127.16.48.1:60413)
0619 16:55:31.947944 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:31.947995 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:31.949806 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:31.950676 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:31.950686 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:31.951254 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:31.951882 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:31.951901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:31.953627 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:31.953636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:31.955598 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:31.955612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:31.956888 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:31.956893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:31.957116 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:31.957936 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:31.957967 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:31.960652 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:31.964208 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:31.964218 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:31.964239 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:31.964717 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:31.965140 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:31.965145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:31.965149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:31.965359 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:31.966011 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:31.966023 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:31.966480 (+ 457us) client_negotiation.cc:769] Sending connection context
0619 16:55:31.966845 (+ 365us) client_negotiation.cc:240] Negotiation successful
0619 16:55:31.967385 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":406,"threads_started":1}
I20260619 16:55:31.970862 19214 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46597
I20260619 16:55:31.971246 19214 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:31.972446 19214 heartbeater.cc:511] Master 127.16.48.62:46597 requested a full tablet report, sending...
I20260619 16:55:31.975360 19025 ts_manager.cc:194] Registered new tserver with Master: 41922d6d9638416cabc2595df50ea0a5 (127.16.48.1:38299)
I20260619 16:55:31.976948 19025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:60413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:55:31.995764 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46597
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:32.343887 19220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:32.344326 19220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:32.344519 19220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:32.352610 19220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:32.352746 19220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:32.352978 19220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:32.363634 19220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:32.365156 19220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:32.366582 19220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:32.381224 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:32.381247 19226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:32.381273 19228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:32.385924 19220 server_base.cc:1034] running on GCE node
I20260619 16:55:32.386656 19220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:32.387780 19220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:32.389031 19220 hybrid_clock.cc:648] HybridClock initialized: now 1781888132388980 us; error 52 us; skew 500 ppm
I20260619 16:55:32.389394 19220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:32 dist-test-slave-3ffd krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888132, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:32.392949 19220 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:55:32.394726 19220 webserver.cc:460] Webserver started at http://127.16.48.2:37319/ using document root <none> and password file <none>
I20260619 16:55:32.395599 19220 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:32.395711 19220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:32.395998 19220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:32.399372 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "ddcb24472f224ff6b6e6da00c7004404"
format_stamp: "Formatted at 2026-06-19 16:55:32 on dist-test-slave-3ffd"
server_key: "edd2485b054dfaf67ea16a0319ba5882"
server_key_iv: "4fad6b21a60a875b0a299f292ad87e5c"
server_key_version: "encryptionkey@0"
I20260619 16:55:32.400277 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "ddcb24472f224ff6b6e6da00c7004404"
format_stamp: "Formatted at 2026-06-19 16:55:32 on dist-test-slave-3ffd"
server_key: "edd2485b054dfaf67ea16a0319ba5882"
server_key_iv: "4fad6b21a60a875b0a299f292ad87e5c"
server_key_version: "encryptionkey@0"
I20260619 16:55:32.406136 19220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:55:32.410125 19235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:32.411744 19220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:55:32.411913 19220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "ddcb24472f224ff6b6e6da00c7004404"
format_stamp: "Formatted at 2026-06-19 16:55:32 on dist-test-slave-3ffd"
server_key: "edd2485b054dfaf67ea16a0319ba5882"
server_key_iv: "4fad6b21a60a875b0a299f292ad87e5c"
server_key_version: "encryptionkey@0"
I20260619 16:55:32.412106 19220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:32.456578 19220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:32.461099 19220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:32.461380 19220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:32.462365 19220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:32.463770 19220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:32.463843 19220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:32.463928 19220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:32.463975 19220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:32.500491 19220 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:37971
I20260619 16:55:32.500659 19347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:37971 every 8 connection(s)
I20260619 16:55:32.502449 19220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:32.507658 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 19220
I20260619 16:55:32.507862 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:32.508443 16576 external_mini_cluster.cc:1428] Setting key c7f862712f67d0dc548b4029339072a8
Jun 19 16:55:32 dist-test-slave-3ffd krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888132, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:32.527602 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:32.505795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:40463 (local address 127.16.48.62:46597)
0619 16:55:32.506226 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:32.506231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:32.507378 (+ 1147us) server_negotiation.cc:407] Connection header received
0619 16:55:32.510064 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:32.510068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:32.510181 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:32.510316 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:32.512170 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:32.513440 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:32.515229 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:32.515999 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:32.516187 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:32.521009 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:32.521015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:32.521018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:32.521056 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:32.523746 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:32.524995 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:32.525000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:32.525003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:32.525153 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:32.525748 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:32.525751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:32.525753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:32.526029 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:32.526139 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:32.527142 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0619 16:55:32.527332 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
I20260619 16:55:32.529274 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:32.506437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46597 (local address 127.16.48.2:40463)
0619 16:55:32.507169 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:32.507235 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:32.509732 (+ 2497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:32.510642 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:32.510656 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:32.511259 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:32.511912 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:32.511933 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:32.513656 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:32.513660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:32.515040 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:32.515054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:32.516323 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:32.516333 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:32.516718 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:32.517671 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:32.517708 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:32.520695 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:32.524013 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:32.524028 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:32.524054 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:32.524846 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:32.525289 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:32.525297 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:32.525304 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:32.525623 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:32.526174 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:32.526184 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:32.526705 (+ 521us) client_negotiation.cc:769] Sending connection context
0619 16:55:32.527059 (+ 354us) client_negotiation.cc:240] Negotiation successful
0619 16:55:32.527698 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":661,"threads_started":1}
I20260619 16:55:32.530880 19348 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46597
I20260619 16:55:32.531226 19348 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
I20260619 16:55:32.532172 19348 heartbeater.cc:511] Master 127.16.48.62:46597 requested a full tablet report, sending...
I20260619 16:55:32.533854 19025 ts_manager.cc:194] Registered new tserver with Master: ddcb24472f224ff6b6e6da00c7004404 (127.16.48.2:37971)
I20260619 16:55:32.534687 19025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:40463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:55:32.556394 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46597
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:32.912600 19354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:32.912921 19354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:32.913028 19354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:32.920630 19354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:32.920765 19354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:32.920902 19354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:32.931285 19354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:32.932753 19354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:32.934159 19354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:32.950737 19362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:32.950736 19360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:32.950778 19359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:32.951208 19354 server_base.cc:1034] running on GCE node
I20260619 16:55:32.952347 19354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:32.953486 19354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:32.954722 19354 hybrid_clock.cc:648] HybridClock initialized: now 1781888132954669 us; error 37 us; skew 500 ppm
I20260619 16:55:32.955065 19354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:32 dist-test-slave-3ffd krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888132, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:32.958415 19354 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:55:32.960309 19354 webserver.cc:460] Webserver started at http://127.16.48.3:44633/ using document root <none> and password file <none>
I20260619 16:55:32.961099 19354 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:32.961256 19354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:32.961652 19354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:32.964883 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "d86707a5603b4355a333441160a56196"
format_stamp: "Formatted at 2026-06-19 16:55:32 on dist-test-slave-3ffd"
server_key: "54762286f3110ec72ca6ed0900253b83"
server_key_iv: "03051fe8908d7dcfcd3e53c3fdccad06"
server_key_version: "encryptionkey@0"
I20260619 16:55:32.965775 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "d86707a5603b4355a333441160a56196"
format_stamp: "Formatted at 2026-06-19 16:55:32 on dist-test-slave-3ffd"
server_key: "54762286f3110ec72ca6ed0900253b83"
server_key_iv: "03051fe8908d7dcfcd3e53c3fdccad06"
server_key_version: "encryptionkey@0"
I20260619 16:55:32.971393 19354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:55:32.975399 19369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:32.977159 19354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:32.977401 19354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "d86707a5603b4355a333441160a56196"
format_stamp: "Formatted at 2026-06-19 16:55:32 on dist-test-slave-3ffd"
server_key: "54762286f3110ec72ca6ed0900253b83"
server_key_iv: "03051fe8908d7dcfcd3e53c3fdccad06"
server_key_version: "encryptionkey@0"
I20260619 16:55:32.977643 19354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:32.980202 19214 heartbeater.cc:503] Master 127.16.48.62:46597 was elected leader, sending a full tablet report...
I20260619 16:55:33.002424 19354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:33.007370 19354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:33.007725 19354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:33.008836 19354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:33.010334 19354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:33.010447 19354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:33.010599 19354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:33.010672 19354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:33.049259 19354 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:39229
I20260619 16:55:33.049297 19481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:39229 every 8 connection(s)
I20260619 16:55:33.050637 19354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:33.052249 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 19354
I20260619 16:55:33.052464 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:33.052901 16576 external_mini_cluster.cc:1428] Setting key 7e5c08acd93b24ed068cc7232a0f11a9
Jun 19 16:55:33 dist-test-slave-3ffd krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888132, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:33.073581 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:33.053494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:47739 (local address 127.16.48.62:46597)
0619 16:55:33.054196 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:33.054202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:33.055357 (+ 1155us) server_negotiation.cc:407] Connection header received
0619 16:55:33.057499 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:33.057504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:33.057595 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:33.057726 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:33.059390 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:33.060263 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:33.062025 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:33.062470 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:33.062613 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:33.066665 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:33.066672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:33.066677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:33.066734 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:33.070128 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:33.071032 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:33.071036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:33.071038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:33.071172 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:33.071747 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:33.071752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:33.071754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:33.072054 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:33.072315 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:33.073097 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 16:55:33.073292 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":456,"threads_started":1}
I20260619 16:55:33.075063 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:33.054347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46597 (local address 127.16.48.3:47739)
0619 16:55:33.055211 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:33.055257 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:33.057192 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:33.057990 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:33.058000 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:33.058568 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:33.059190 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:33.059211 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:33.060439 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:33.060443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:33.061863 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:33.061877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:33.062721 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:33.062728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:33.062984 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:33.063822 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:33.063852 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:33.066369 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:33.070332 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:33.070341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:33.070358 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:33.070902 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:33.071333 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:33.071337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:33.071339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:33.071582 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:33.072281 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:33.072288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:33.072715 (+ 427us) client_negotiation.cc:769] Sending connection context
0619 16:55:33.073004 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 16:55:33.073459 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":350,"threads_started":1}
I20260619 16:55:33.076803 19482 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46597
I20260619 16:55:33.077128 19482 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:33.077966 19482 heartbeater.cc:511] Master 127.16.48.62:46597 requested a full tablet report, sending...
I20260619 16:55:33.079878 19025 ts_manager.cc:194] Registered new tserver with Master: d86707a5603b4355a333441160a56196 (127.16.48.3:39229)
I20260619 16:55:33.081041 19025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:47739
I20260619 16:55:33.082159 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:55:33.115598 19484 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:55:33.113065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.16.48.62:46597)
0619 16:55:33.113285 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:33.113289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:33.113762 (+ 473us) server_negotiation.cc:407] Connection header received
0619 16:55:33.113900 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:33.113904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:33.113972 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:33.114087 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:33.115329 (+ 1242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46888: BlockingRecv error: recv got EOF from 127.0.0.1:46888 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 16:55:33.119019 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19086
W20260619 16:55:33.122546 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19086: Not authorized: Client connection negotiation failed: client connection to 127.16.48.1:38299: server requires authentication, but client does not have Kerberos credentials available
I20260619 16:55:33.122704 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 19086
I20260619 16:55:33.139405 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19220
W20260619 16:55:33.143622 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19220: Not authorized: Client connection negotiation failed: client connection to 127.16.48.2:37971: server requires authentication, but client does not have Kerberos credentials available
W20260619 16:55:33.143911 19495 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:55:33.140765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.16.48.2:37971)
0619 16:55:33.141485 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:33.141524 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:33.141554 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:55:33.141617 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:33.141623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:33.141853 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:33.141992 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:33.143560 (+ 1568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51652: BlockingRecv error: recv got EOF from 127.0.0.1:51652 (error 108)
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":445,"threads_started":1}
I20260619 16:55:33.144014 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 19220
I20260619 16:55:33.164825 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19354
W20260619 16:55:33.168083 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19354: Not authorized: Client connection negotiation failed: client connection to 127.16.48.3:39229: server requires authentication, but client does not have Kerberos credentials available
I20260619 16:55:33.168277 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 19354
I20260619 16:55:33.189086 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 18993
I20260619 16:55:33.190501 19055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:33.297312 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 18993
2026-06-19T16:55:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3748 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:33 dist-test-slave-3ffd krb5kdc[19500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:33 dist-test-slave-3ffd krb5kdc[19500](info): set up 2 sockets
Jun 19 16:55:33 dist-test-slave-3ffd krb5kdc[19500](info): commencing operation
krb5kdc: starting...
W20260619 16:55:34.436880 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.008s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:34 dist-test-slave-3ffd krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:34.649190 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:40525
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:40525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:35.004961 19513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:35.005306 19513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:35.005491 19513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:35.013382 19513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:35.013517 19513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:35.013634 19513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:35.013741 19513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:35.013825 19513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:35.024451 19513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:40525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:40525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:35.025947 19513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:35.027451 19513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:35.044286 19518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:35.044312 19521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:35.044267 19519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:35.045542 19513 server_base.cc:1034] running on GCE node
I20260619 16:55:35.046402 19513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:35.047994 19513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:35.049225 19513 hybrid_clock.cc:648] HybridClock initialized: now 1781888135049149 us; error 63 us; skew 500 ppm
I20260619 16:55:35.049664 19513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:35 dist-test-slave-3ffd krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888135, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:35.057168 19513 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:35.058985 19513 webserver.cc:460] Webserver started at http://127.16.48.62:36711/ using document root <none> and password file <none>
I20260619 16:55:35.059757 19513 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:35.059906 19513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:35.060315 19513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:35.063591 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "3df99e8cd5bb4f418a7937536914a990"
format_stamp: "Formatted at 2026-06-19 16:55:35 on dist-test-slave-3ffd"
server_key: "fdc43676d5ec7d137df54efbcac4275a"
server_key_iv: "485198a0ecfdc63cb0555975ccbb96a5"
server_key_version: "encryptionkey@0"
I20260619 16:55:35.064508 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "3df99e8cd5bb4f418a7937536914a990"
format_stamp: "Formatted at 2026-06-19 16:55:35 on dist-test-slave-3ffd"
server_key: "fdc43676d5ec7d137df54efbcac4275a"
server_key_iv: "485198a0ecfdc63cb0555975ccbb96a5"
server_key_version: "encryptionkey@0"
I20260619 16:55:35.070570 19513 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:55:35.074595 19528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:35.076277 19513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:55:35.076494 19513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "3df99e8cd5bb4f418a7937536914a990"
format_stamp: "Formatted at 2026-06-19 16:55:35 on dist-test-slave-3ffd"
server_key: "fdc43676d5ec7d137df54efbcac4275a"
server_key_iv: "485198a0ecfdc63cb0555975ccbb96a5"
server_key_version: "encryptionkey@0"
I20260619 16:55:35.076718 19513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:35.108203 19513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:35.113217 19513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:35.113562 19513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:35.136579 19513 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:40525
I20260619 16:55:35.136708 19579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:40525 every 8 connection(s)
I20260619 16:55:35.138180 19513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:35.140666 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 19513
I20260619 16:55:35.140883 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:35.141726 16576 external_mini_cluster.cc:1428] Setting key d7ee1c5cffc6573957df64d1e0ee0d70
I20260619 16:55:35.143388 19580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:55:35 dist-test-slave-3ffd krb5kdc[19500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:35.158946 19580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990: Bootstrap starting.
I20260619 16:55:35.162521 19580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:35.163697 19580 log.cc:824] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:35.167554 19580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990: No bootstrap required, opened a new log
I20260619 16:55:35.168200 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:35.144081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.16.48.62:40525)
0619 16:55:35.145098 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:35.145111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:35.145161 (+ 50us) server_negotiation.cc:407] Connection header received
0619 16:55:35.147066 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:35.147096 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:35.147628 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:35.148185 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:35.150201 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:35.151726 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:35.153625 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:35.154399 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:35.154573 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:35.159460 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:35.159470 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:35.159488 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:35.159534 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:35.162994 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:35.163802 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:35.163810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:35.163818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:35.164046 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:35.164571 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:35.164576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:35.164578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:35.165325 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:35.165644 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:35.166122 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 16:55:35.166637 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":419,"threads_started":1}
I20260619 16:55:35.173688 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:55:35.179527 19580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df99e8cd5bb4f418a7937536914a990" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40525 } }
I20260619 16:55:35.180223 19580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:35.180349 19580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df99e8cd5bb4f418a7937536914a990, State: Initialized, Role: FOLLOWER
I20260619 16:55:35.181173 19580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df99e8cd5bb4f418a7937536914a990" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40525 } }
I20260619 16:55:35.181522 19580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:35.181679 19580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:35.181890 19580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:35.215704 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:35.208095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.16.48.62:40525)
0619 16:55:35.208388 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:35.208398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:35.208853 (+ 455us) server_negotiation.cc:407] Connection header received
0619 16:55:35.209011 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:35.209017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:35.209086 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:35.209197 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:35.210256 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:35.210998 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:35.212463 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:35.212887 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:35.213006 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:35.214198 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:35.214203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:35.214206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:35.214868 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:35.214928 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:35.214958 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:35.215050 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:35.215226 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 16:55:35.215310 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 16:55:35.232285 19580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df99e8cd5bb4f418a7937536914a990" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40525 } }
I20260619 16:55:35.232962 19580 leader_election.cc:302] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3df99e8cd5bb4f418a7937536914a990; no voters:
I20260619 16:55:35.234426 19580 leader_election.cc:288] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:35.234536 19592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:35.235836 19592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [term 1 LEADER]: Becoming Leader. State: Replica: 3df99e8cd5bb4f418a7937536914a990, State: Running, Role: LEADER
I20260619 16:55:35.236553 19592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df99e8cd5bb4f418a7937536914a990" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40525 } }
I20260619 16:55:35.237836 19580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:35.240815 19593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3df99e8cd5bb4f418a7937536914a990. Latest consensus state: current_term: 1 leader_uuid: "3df99e8cd5bb4f418a7937536914a990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df99e8cd5bb4f418a7937536914a990" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40525 } } }
I20260619 16:55:35.241395 19593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:35.240906 19594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3df99e8cd5bb4f418a7937536914a990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df99e8cd5bb4f418a7937536914a990" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40525 } } }
I20260619 16:55:35.241950 19594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:35.242498 19601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:35.248617 19601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:35.259483 19601 catalog_manager.cc:1347] Generated new cluster ID: e18ec87bd837410eaaf7b93544a0928e
I20260619 16:55:35.259632 19601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:35.280603 19601 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:35.282203 19601 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:35.291904 19601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3df99e8cd5bb4f418a7937536914a990: Generated new TSK 0
I20260619 16:55:35.292953 19601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:55:35.307212 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19513
I20260619 16:55:35.308614 19575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:35.407230 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 19513
2026-06-19T16:55:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2105 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:36 dist-test-slave-3ffd krb5kdc[19614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:36 dist-test-slave-3ffd krb5kdc[19614](info): set up 2 sockets
Jun 19 16:55:36 dist-test-slave-3ffd krb5kdc[19614](info): commencing operation
krb5kdc: starting...
W20260619 16:55:37.732918 16576 mini_kdc.cc:121] Time spent starting KDC: real 2.287s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:37 dist-test-slave-3ffd krb5kdc[19614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:37.953433 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:37097
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:38.307233 19627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:38.307560 19627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:38.307663 19627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:38.315086 19627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:38.315219 19627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:38.315268 19627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:38.315318 19627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:38.315366 19627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:38.325755 19627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:37097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:38.327101 19627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:38.328437 19627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:38.341418 19632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:38.341639 19633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:38.341503 19635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:38.342404 19627 server_base.cc:1034] running on GCE node
I20260619 16:55:38.343317 19627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:38.345023 19627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:38.346294 19627 hybrid_clock.cc:648] HybridClock initialized: now 1781888138346228 us; error 47 us; skew 500 ppm
I20260619 16:55:38.346758 19627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:38 dist-test-slave-3ffd krb5kdc[19614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:38.350414 19627 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:38.352373 19627 webserver.cc:460] Webserver started at http://127.16.48.62:42375/ using document root <none> and password file <none>
I20260619 16:55:38.353137 19627 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:38.353260 19627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:38.353571 19627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:38.357187 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "ed8415664bee4d9dbd0e6bfbfb154670"
format_stamp: "Formatted at 2026-06-19 16:55:38 on dist-test-slave-3ffd"
server_key: "e39d698a362b9bda1ae3580b3fd4eec9"
server_key_iv: "2464e27476659aef5a04598c10476f14"
server_key_version: "encryptionkey@0"
I20260619 16:55:38.358067 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "ed8415664bee4d9dbd0e6bfbfb154670"
format_stamp: "Formatted at 2026-06-19 16:55:38 on dist-test-slave-3ffd"
server_key: "e39d698a362b9bda1ae3580b3fd4eec9"
server_key_iv: "2464e27476659aef5a04598c10476f14"
server_key_version: "encryptionkey@0"
I20260619 16:55:38.364558 19627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 16:55:38.368893 19642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:38.370605 19627 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:55:38.370888 19627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "ed8415664bee4d9dbd0e6bfbfb154670"
format_stamp: "Formatted at 2026-06-19 16:55:38 on dist-test-slave-3ffd"
server_key: "e39d698a362b9bda1ae3580b3fd4eec9"
server_key_iv: "2464e27476659aef5a04598c10476f14"
server_key_version: "encryptionkey@0"
I20260619 16:55:38.371135 19627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:38.390098 19627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:38.395051 19627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:38.395402 19627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:38.415143 19627 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:37097
I20260619 16:55:38.415175 19693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:37097 every 8 connection(s)
I20260619 16:55:38.416661 19627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:38.420801 19694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:38.425091 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 19627
I20260619 16:55:38.425336 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:38.426205 16576 external_mini_cluster.cc:1428] Setting key c9b743a01c01b1f030c9722115fec4e3
I20260619 16:55:38.433818 19694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670: Bootstrap starting.
I20260619 16:55:38.437875 19694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:38.439260 19694 log.cc:824] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:55:38 dist-test-slave-3ffd krb5kdc[19614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:38.443152 19694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670: No bootstrap required, opened a new log
I20260619 16:55:38.451656 19694 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8415664bee4d9dbd0e6bfbfb154670" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37097 } }
I20260619 16:55:38.452111 19694 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:38.452293 19694 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8415664bee4d9dbd0e6bfbfb154670, State: Initialized, Role: FOLLOWER
I20260619 16:55:38.452792 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:38.428568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.16.48.62:37097)
0619 16:55:38.429454 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:38.429466 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:38.429510 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:55:38.431905 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:38.431933 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:38.432492 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:38.432880 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:38.434721 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:38.436083 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:38.438239 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:38.439012 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:38.439168 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:38.444280 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:38.444290 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:38.444308 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:38.444355 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:38.447829 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:38.448806 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:38.448815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:38.448823 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:38.449020 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:38.449492 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:38.449496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:38.449498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:38.450183 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:38.450553 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:38.451034 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 16:55:38.451498 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":401,"threads_started":1}
I20260619 16:55:38.453083 19694 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8415664bee4d9dbd0e6bfbfb154670" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37097 } }
I20260619 16:55:38.453418 19694 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:38.453624 19694 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:38.453833 19694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:38.456612 19694 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8415664bee4d9dbd0e6bfbfb154670" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37097 } }
I20260619 16:55:38.457145 19694 leader_election.cc:302] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed8415664bee4d9dbd0e6bfbfb154670; no voters:
I20260619 16:55:38.457800 19694 leader_election.cc:288] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:38.457967 19699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:38.459360 19699 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [term 1 LEADER]: Becoming Leader. State: Replica: ed8415664bee4d9dbd0e6bfbfb154670, State: Running, Role: LEADER
I20260619 16:55:38.459923 19699 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8415664bee4d9dbd0e6bfbfb154670" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37097 } }
I20260619 16:55:38.460456 19694 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:38.463349 19701 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8415664bee4d9dbd0e6bfbfb154670. Latest consensus state: current_term: 1 leader_uuid: "ed8415664bee4d9dbd0e6bfbfb154670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8415664bee4d9dbd0e6bfbfb154670" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37097 } } }
I20260619 16:55:38.463840 19701 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:38.464356 19700 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed8415664bee4d9dbd0e6bfbfb154670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8415664bee4d9dbd0e6bfbfb154670" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37097 } } }
I20260619 16:55:38.464741 19700 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:38.466475 19709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:38.471347 19709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:38.481366 19709 catalog_manager.cc:1347] Generated new cluster ID: 5ea9f360f5ac44ed967e8bebee73c094
I20260619 16:55:38.481501 19709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:38.493274 19709 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:38.494853 19709 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:38.505046 19709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed8415664bee4d9dbd0e6bfbfb154670: Generated new TSK 0
I20260619 16:55:38.506053 19709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:55:38.556578 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37097
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:38.919351 19720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:38.919678 19720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:38.919848 19720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:38.928105 19720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:38.928299 19720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:38.928483 19720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:38.939018 19720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:38.940533 19720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:38.942106 19720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:38.957201 19725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:38.957217 19728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:38.957341 19726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:38.958110 19720 server_base.cc:1034] running on GCE node
I20260619 16:55:38.959146 19720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:38.960453 19720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:38.961859 19720 hybrid_clock.cc:648] HybridClock initialized: now 1781888138961765 us; error 79 us; skew 500 ppm
I20260619 16:55:38.962241 19720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:38 dist-test-slave-3ffd krb5kdc[19614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:38.966250 19720 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:55:38.968302 19720 webserver.cc:460] Webserver started at http://127.16.48.1:33133/ using document root <none> and password file <none>
I20260619 16:55:38.969062 19720 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:38.969187 19720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:38.969467 19720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:38.973233 19720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "4d93a72b941a4ef9b76995a7c735d13c"
format_stamp: "Formatted at 2026-06-19 16:55:38 on dist-test-slave-3ffd"
server_key: "5551ef71909efc427030e44f44abb5b9"
server_key_iv: "2374501bdcf6d8b02dd4eded99fb0bf0"
server_key_version: "encryptionkey@0"
I20260619 16:55:38.974115 19720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "4d93a72b941a4ef9b76995a7c735d13c"
format_stamp: "Formatted at 2026-06-19 16:55:38 on dist-test-slave-3ffd"
server_key: "5551ef71909efc427030e44f44abb5b9"
server_key_iv: "2374501bdcf6d8b02dd4eded99fb0bf0"
server_key_version: "encryptionkey@0"
I20260619 16:55:38.980335 19720 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 16:55:38.985083 19735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:38.986871 19720 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:55:38.987155 19720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "4d93a72b941a4ef9b76995a7c735d13c"
format_stamp: "Formatted at 2026-06-19 16:55:38 on dist-test-slave-3ffd"
server_key: "5551ef71909efc427030e44f44abb5b9"
server_key_iv: "2374501bdcf6d8b02dd4eded99fb0bf0"
server_key_version: "encryptionkey@0"
I20260619 16:55:38.987385 19720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:39.007959 19720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:39.013350 19720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:39.013655 19720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:39.014680 19720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:39.016100 19720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:39.016265 19720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 16:55:39.016363 19720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:39.016408 19720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:39.051590 19720 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:34265
I20260619 16:55:39.051684 19847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:34265 every 8 connection(s)
I20260619 16:55:39.052974 19720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:39.058987 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 19720
I20260619 16:55:39.059216 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:39.060061 16576 external_mini_cluster.cc:1428] Setting key 7f7bc55bbab4d6685a1ace656e819f93
Jun 19 16:55:39 dist-test-slave-3ffd krb5kdc[19614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:39.077312 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:39.055805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:46895 (local address 127.16.48.62:37097)
0619 16:55:39.056311 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:39.056316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:39.057402 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 16:55:39.059835 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:39.059840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:39.059929 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:39.060064 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:39.062004 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:39.063072 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:39.064997 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:39.065783 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:39.065960 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:39.070360 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:39.070370 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:39.070376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:39.070430 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:39.073358 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:39.074567 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:39.074572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:39.074574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:39.074718 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:39.075305 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:39.075309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:39.075311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:39.075613 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:39.075786 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:39.076669 (+ 883us) server_negotiation.cc:299] Negotiation successful
0619 16:55:39.076913 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":312,"threads_started":1}
I20260619 16:55:39.078413 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:39.056416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37097 (local address 127.16.48.1:46895)
0619 16:55:39.057220 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:39.057268 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:39.059436 (+ 2168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:39.060428 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:39.060442 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:39.061191 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:39.061819 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:39.061837 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:39.063267 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:39.063272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:39.064786 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:39.064800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:39.066073 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:39.066081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:39.066344 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:39.067144 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:39.067170 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:39.069966 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:39.073561 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:39.073576 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:39.073600 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:39.074386 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:39.074862 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:39.074868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:39.074872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:39.075164 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:39.075760 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:39.075769 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:39.076271 (+ 502us) client_negotiation.cc:769] Sending connection context
0619 16:55:39.076567 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 16:55:39.077170 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 16:55:39.080272 19848 heartbeater.cc:347] Connected to a master server at 127.16.48.62:37097
I20260619 16:55:39.080615 19848 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:39.081426 19848 heartbeater.cc:511] Master 127.16.48.62:37097 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:55:39.084091 19659 ts_manager.cc:194] Registered new tserver with Master: 4d93a72b941a4ef9b76995a7c735d13c (127.16.48.1:34265)
I20260619 16:55:39.086407 19659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:46895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:55:39.107784 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37097
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:39.457641 19854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:39.457945 19854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:39.458043 19854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:39.465803 19854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:39.465932 19854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:39.466074 19854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:39.477252 19854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:39.478624 19854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:39.480002 19854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:39.494616 19862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:39.494704 19860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:39.494680 19859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:39.495496 19854 server_base.cc:1034] running on GCE node
I20260619 16:55:39.496273 19854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:39.497334 19854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:39.498584 19854 hybrid_clock.cc:648] HybridClock initialized: now 1781888139498532 us; error 47 us; skew 500 ppm
I20260619 16:55:39.498944 19854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:39 dist-test-slave-3ffd krb5kdc[19614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888139, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:39.502436 19854 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:55:39.504101 19854 webserver.cc:460] Webserver started at http://127.16.48.2:41909/ using document root <none> and password file <none>
I20260619 16:55:39.504935 19854 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:39.505050 19854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:39.505497 19854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:39.508839 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "1e0e5dd177744173a2af135a9d3d77c9"
format_stamp: "Formatted at 2026-06-19 16:55:39 on dist-test-slave-3ffd"
server_key: "4810d2b24587cc0cabb34dbcc4ffa9c4"
server_key_iv: "93651a607d32141acc1713cd581dfe47"
server_key_version: "encryptionkey@0"
I20260619 16:55:39.509660 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "1e0e5dd177744173a2af135a9d3d77c9"
format_stamp: "Formatted at 2026-06-19 16:55:39 on dist-test-slave-3ffd"
server_key: "4810d2b24587cc0cabb34dbcc4ffa9c4"
server_key_iv: "93651a607d32141acc1713cd581dfe47"
server_key_version: "encryptionkey@0"
I20260619 16:55:39.515350 19854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:55:39.519225 19869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:39.520886 19854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:39.521056 19854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "1e0e5dd177744173a2af135a9d3d77c9"
format_stamp: "Formatted at 2026-06-19 16:55:39 on dist-test-slave-3ffd"
server_key: "4810d2b24587cc0cabb34dbcc4ffa9c4"
server_key_iv: "93651a607d32141acc1713cd581dfe47"
server_key_version: "encryptionkey@0"
I20260619 16:55:39.521238 19854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:39.539299 19854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:39.544236 19854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:39.544535 19854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:39.545533 19854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:39.546958 19854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:39.547039 19854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:39.547133 19854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:39.547179 19854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:39.582885 19854 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:35005
I20260619 16:55:39.582928 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:35005 every 8 connection(s)
I20260619 16:55:39.584355 19854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:39.589301 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 19854
I20260619 16:55:39.589473 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:39.589907 16576 external_mini_cluster.cc:1428] Setting key 623af8986fade62681996796eed583ee
Jun 19 16:55:39 dist-test-slave-3ffd krb5kdc[19614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888139, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:39.608151 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:39.587323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:54757 (local address 127.16.48.62:37097)
0619 16:55:39.587785 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:39.587789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:39.588938 (+ 1149us) server_negotiation.cc:407] Connection header received
0619 16:55:39.591154 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:39.591159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:39.591263 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:39.591422 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:39.593509 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:39.594256 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:39.595936 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:39.596442 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:39.596600 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:39.601548 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:39.601555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:39.601558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:39.601596 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:39.604441 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:39.605562 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:39.605567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:39.605569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:39.605709 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:39.606230 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:39.606235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:39.606237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:39.606524 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:39.606691 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:39.607490 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 16:55:39.607738 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":253,"threads_started":1}
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
I20260619 16:55:39.609354 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:39.587837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37097 (local address 127.16.48.2:54757)
0619 16:55:39.588746 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:39.588790 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:39.590834 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:39.591869 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:39.591881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:39.592658 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:39.593305 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:39.593325 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:39.594452 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:39.594456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:39.595764 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:39.595778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:39.596718 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:39.596725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:39.596985 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:39.597773 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:39.597826 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:39.601224 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:39.604805 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:39.604814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:39.604831 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:39.605380 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:39.605841 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:39.605847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:39.605851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:39.606088 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:39.606668 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:39.606678 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:39.607095 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 16:55:39.607389 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 16:55:39.608017 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":423,"threads_started":1}
I20260619 16:55:39.611162 19982 heartbeater.cc:347] Connected to a master server at 127.16.48.62:37097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
I20260619 16:55:39.611557 19982 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:39.612636 19982 heartbeater.cc:511] Master 127.16.48.62:37097 requested a full tablet report, sending...
I20260619 16:55:39.614547 19659 ts_manager.cc:194] Registered new tserver with Master: 1e0e5dd177744173a2af135a9d3d77c9 (127.16.48.2:35005)
I20260619 16:55:39.615438 19659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:54757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:55:39.639145 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37097
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:39.999118 19988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:39.999432 19988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:39.999528 19988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:40.007170 19988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:40.007318 19988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:40.007436 19988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:40.017732 19988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:40.019110 19988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:40.020602 19988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:40.035339 19994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:40.035311 19993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:40.035285 19996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:40.036607 19988 server_base.cc:1034] running on GCE node
I20260619 16:55:40.037295 19988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:40.038378 19988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:40.039633 19988 hybrid_clock.cc:648] HybridClock initialized: now 1781888140039572 us; error 56 us; skew 500 ppm
I20260619 16:55:40.039999 19988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[19614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888140, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:40.043594 19988 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:55:40.045336 19988 webserver.cc:460] Webserver started at http://127.16.48.3:37375/ using document root <none> and password file <none>
I20260619 16:55:40.046123 19988 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:40.046281 19988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:40.046631 19988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:40.050107 19988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "a0b8b139a4d24eecaef0399592214159"
format_stamp: "Formatted at 2026-06-19 16:55:40 on dist-test-slave-3ffd"
server_key: "1d04717b88054a77f6759f1601337608"
server_key_iv: "38428ea312d8ed067898ff281fddf048"
server_key_version: "encryptionkey@0"
I20260619 16:55:40.051386 19988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "a0b8b139a4d24eecaef0399592214159"
format_stamp: "Formatted at 2026-06-19 16:55:40 on dist-test-slave-3ffd"
server_key: "1d04717b88054a77f6759f1601337608"
server_key_iv: "38428ea312d8ed067898ff281fddf048"
server_key_version: "encryptionkey@0"
I20260619 16:55:40.057503 19988 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 16:55:40.061748 20003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:40.063282 19988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:40.063438 19988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "a0b8b139a4d24eecaef0399592214159"
format_stamp: "Formatted at 2026-06-19 16:55:40 on dist-test-slave-3ffd"
server_key: "1d04717b88054a77f6759f1601337608"
server_key_iv: "38428ea312d8ed067898ff281fddf048"
server_key_version: "encryptionkey@0"
I20260619 16:55:40.063632 19988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:40.090219 19848 heartbeater.cc:503] Master 127.16.48.62:37097 was elected leader, sending a full tablet report...
I20260619 16:55:40.097934 19988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:40.102526 19988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:40.102808 19988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:40.103832 19988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:40.105388 19988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:40.105465 19988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:40.105556 19988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:40.105598 19988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:40.140591 19988 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:45753
I20260619 16:55:40.140654 20115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:45753 every 8 connection(s)
I20260619 16:55:40.142036 19988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:40.151696 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 19988
I20260619 16:55:40.151901 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:40.152546 16576 external_mini_cluster.cc:1428] Setting key 372e5b51a22f605ddc5fb53c2b195c22
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[19614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888140, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:40.165551 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:40.144933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:36017 (local address 127.16.48.62:37097)
0619 16:55:40.145425 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:40.145430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:40.146591 (+ 1161us) server_negotiation.cc:407] Connection header received
0619 16:55:40.148596 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:40.148601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:40.148690 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:40.148819 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:40.150592 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.151450 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.154156 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.154629 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.154782 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:40.158913 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:40.158919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:40.158923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:40.158961 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:40.161994 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.162905 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.162910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.162912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.163060 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.163596 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.163599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.163601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.163887 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:40.164010 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:40.164955 (+ 945us) server_negotiation.cc:299] Negotiation successful
0619 16:55:40.165246 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
I20260619 16:55:40.166675 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:40.145616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37097 (local address 127.16.48.3:36017)
0619 16:55:40.146388 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:40.146440 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:40.148274 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:40.149088 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:40.149097 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:40.149687 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:40.150365 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:40.150386 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.151665 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.151671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:40.153960 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:40.153996 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.155081 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.155086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:40.155312 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:40.156180 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:40.156220 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:40.158664 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:40.162167 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:40.162177 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:40.162194 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:40.162725 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:40.163248 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:40.163252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:40.163255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:40.163476 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:40.164006 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:40.164012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:40.164469 (+ 457us) client_negotiation.cc:769] Sending connection context
0619 16:55:40.164820 (+ 351us) client_negotiation.cc:240] Negotiation successful
0619 16:55:40.165286 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":304,"threads_started":1}
I20260619 16:55:40.168540 20116 heartbeater.cc:347] Connected to a master server at 127.16.48.62:37097
I20260619 16:55:40.168962 20116 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:40.169855 20116 heartbeater.cc:511] Master 127.16.48.62:37097 requested a full tablet report, sending...
I20260619 16:55:40.171507 19659 ts_manager.cc:194] Registered new tserver with Master: a0b8b139a4d24eecaef0399592214159 (127.16.48.3:45753)
I20260619 16:55:40.172495 19659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:36017
I20260619 16:55:40.181999 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[19614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[19614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:40.234508 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:40.219808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.16.48.62:37097)
0619 16:55:40.220000 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:40.220005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:40.220606 (+ 601us) server_negotiation.cc:407] Connection header received
0619 16:55:40.220766 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:40.220770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:40.220839 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:40.220935 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:40.222262 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.222987 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.224271 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.224773 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.224870 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:40.229179 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:40.229185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:40.229188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:40.229246 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:40.232007 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.232727 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.232731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.232734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.232854 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.233264 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.233267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.233269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.233588 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:40.233765 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:40.234091 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 16:55:40.234273 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 16:55:40.235067 19659 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34662
I20260619 16:55:40.236917 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19720
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[19614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:40.253381 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:40.237978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.16.48.1:34265)
0619 16:55:40.238610 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:40.238618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:40.238643 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:55:40.238701 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:40.238708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:40.238923 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:40.239054 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:40.240297 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.241338 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.242577 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.243155 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.243267 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:40.247381 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:40.247396 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:40.247413 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:40.247454 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:40.250546 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.251290 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.251297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.251300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.251428 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.251805 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.251808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.251809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.252240 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:40.252484 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:40.252724 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 16:55:40.252937 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":325,"threads_started":1}
W20260619 16:55:40.254767 19843 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33142
W20260619 16:55:40.256063 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19720: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:55:40.256352 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 19720
I20260619 16:55:40.272835 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19854
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[19614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:55:40.290063 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:40.273762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.16.48.2:35005)
0619 16:55:40.274363 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:40.274370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:40.274398 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:55:40.274469 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:40.274475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:40.274706 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:40.274922 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:40.276332 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.277402 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.278656 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.279215 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.279317 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:40.283665 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:40.283680 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:40.283717 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:40.283759 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:40.286936 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.287732 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.287741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.287743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.287878 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.288296 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.288299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.288301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.288677 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:40.288917 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:40.289414 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 16:55:40.289740 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":316,"threads_started":1}
W20260619 16:55:40.291492 19977 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54766
W20260619 16:55:40.292743 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19854: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:55:40.292889 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 19854
I20260619 16:55:40.308746 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19988
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[19614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:55:40.325948 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:40.309696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.16.48.3:45753)
0619 16:55:40.310399 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:40.310410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:40.310454 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:55:40.310557 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:40.310581 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:40.310953 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:40.311136 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:40.312657 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.313721 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.314996 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:40.315578 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:40.315683 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:40.320262 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:40.320278 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:40.320296 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:40.320340 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:40.323092 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.323840 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.323848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.323850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.323982 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:40.324461 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:40.324465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:40.324467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:40.324855 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:40.325086 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:40.325425 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 16:55:40.325637 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":334,"threads_started":1}
W20260619 16:55:40.327255 20111 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45288
W20260619 16:55:40.328464 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19988: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:55:40.328653 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 19988
I20260619 16:55:40.344277 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 19627
I20260619 16:55:40.345602 19689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:40.448021 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 19627
2026-06-19T16:55:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5045 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[20134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[20134](info): set up 2 sockets
Jun 19 16:55:40 dist-test-slave-3ffd krb5kdc[20134](info): commencing operation
krb5kdc: starting...
W20260619 16:55:41.545178 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:41 dist-test-slave-3ffd krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:41.759730 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:37141
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:42.105068 20147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:42.105412 20147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:42.105504 20147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:42.112852 20147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:42.112979 20147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:42.113025 20147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:42.113078 20147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:42.113123 20147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:42.123261 20147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:42.124704 20147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:42.126104 20147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:42.138840 20152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:42.138780 20155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:42.138806 20153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:42.139899 20147 server_base.cc:1034] running on GCE node
I20260619 16:55:42.140877 20147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:42.142544 20147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:42.143807 20147 hybrid_clock.cc:648] HybridClock initialized: now 1781888142143755 us; error 47 us; skew 500 ppm
I20260619 16:55:42.144251 20147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:42 dist-test-slave-3ffd krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:42.147596 20147 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:42.149398 20147 webserver.cc:460] Webserver started at http://127.16.48.62:36315/ using document root <none> and password file <none>
I20260619 16:55:42.150180 20147 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:42.150344 20147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:42.150691 20147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:42.154011 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "7e3097be53224dfc8c4e1d6184b69b0c"
format_stamp: "Formatted at 2026-06-19 16:55:42 on dist-test-slave-3ffd"
server_key: "77c739902ba2d41666fd25108743f167"
server_key_iv: "fe62f9128c1d296856cc695bbcf3dac9"
server_key_version: "encryptionkey@0"
I20260619 16:55:42.154973 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "7e3097be53224dfc8c4e1d6184b69b0c"
format_stamp: "Formatted at 2026-06-19 16:55:42 on dist-test-slave-3ffd"
server_key: "77c739902ba2d41666fd25108743f167"
server_key_iv: "fe62f9128c1d296856cc695bbcf3dac9"
server_key_version: "encryptionkey@0"
I20260619 16:55:42.161024 20147 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 16:55:42.165186 20162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:42.166899 20147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:42.167140 20147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "7e3097be53224dfc8c4e1d6184b69b0c"
format_stamp: "Formatted at 2026-06-19 16:55:42 on dist-test-slave-3ffd"
server_key: "77c739902ba2d41666fd25108743f167"
server_key_iv: "fe62f9128c1d296856cc695bbcf3dac9"
server_key_version: "encryptionkey@0"
I20260619 16:55:42.167394 20147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:42.198467 20147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:42.204013 20147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:42.204380 20147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:42.224542 20147 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:37141
I20260619 16:55:42.224565 20213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:37141 every 8 connection(s)
I20260619 16:55:42.226140 20147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:42.230369 20214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:42.231220 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 20147
I20260619 16:55:42.231455 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:42.232291 16576 external_mini_cluster.cc:1428] Setting key 5ded13ba0188fe3c4cd70f3aad69db4d
I20260619 16:55:42.244485 20214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c: Bootstrap starting.
Jun 19 16:55:42 dist-test-slave-3ffd krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:42.247994 20214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:42.249142 20214 log.cc:824] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:42.252599 20214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c: No bootstrap required, opened a new log
I20260619 16:55:42.257159 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:42.234833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.16.48.62:37141)
0619 16:55:42.235705 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:42.235720 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:42.235772 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:55:42.237694 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:42.237726 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:42.238252 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:42.238732 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:42.240433 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:42.241861 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:42.243223 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:42.243956 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:42.244201 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:42.248658 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:42.248667 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:42.248686 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:42.248736 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:42.251932 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:42.252812 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:42.252822 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:42.252831 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:42.253063 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:42.253495 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:42.253500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:42.253503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:42.254299 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:42.254621 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:42.254983 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 16:55:42.255513 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":365,"threads_started":1}
I20260619 16:55:42.262769 20214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3097be53224dfc8c4e1d6184b69b0c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37141 } }
I20260619 16:55:42.263186 20214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:42.263268 20214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3097be53224dfc8c4e1d6184b69b0c, State: Initialized, Role: FOLLOWER
I20260619 16:55:42.263885 20214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3097be53224dfc8c4e1d6184b69b0c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37141 } }
I20260619 16:55:42.264106 20214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:42.264281 20214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:42.264415 20214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:42.266932 20214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3097be53224dfc8c4e1d6184b69b0c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37141 } }
I20260619 16:55:42.267469 20214 leader_election.cc:302] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e3097be53224dfc8c4e1d6184b69b0c; no voters:
I20260619 16:55:42.268057 20214 leader_election.cc:288] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:42.268265 20219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:42.268959 20219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [term 1 LEADER]: Becoming Leader. State: Replica: 7e3097be53224dfc8c4e1d6184b69b0c, State: Running, Role: LEADER
I20260619 16:55:42.269631 20219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3097be53224dfc8c4e1d6184b69b0c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37141 } }
I20260619 16:55:42.270251 20214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:42.272645 20221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e3097be53224dfc8c4e1d6184b69b0c. Latest consensus state: current_term: 1 leader_uuid: "7e3097be53224dfc8c4e1d6184b69b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3097be53224dfc8c4e1d6184b69b0c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37141 } } }
I20260619 16:55:42.273102 20221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:42.272975 20220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e3097be53224dfc8c4e1d6184b69b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3097be53224dfc8c4e1d6184b69b0c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37141 } } }
I20260619 16:55:42.273278 20220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:42.274101 20227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:42.280043 20227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:42.290423 20227 catalog_manager.cc:1347] Generated new cluster ID: df96dc15dba442cdabd28707bd391524
I20260619 16:55:42.290589 20227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:42.312547 20227 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:42.313978 20227 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:42.324784 20227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e3097be53224dfc8c4e1d6184b69b0c: Generated new TSK 0
I20260619 16:55:42.325815 20227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:55:42.375636 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37141
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:42.733537 20240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:42.733875 20240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:42.734043 20240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:42.741848 20240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:42.741986 20240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:42.742174 20240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:42.752918 20240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:42.754354 20240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:42.755769 20240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:42.769999 20245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:42.769980 20246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:42.770022 20248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:42.770948 20240 server_base.cc:1034] running on GCE node
I20260619 16:55:42.771699 20240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:42.772866 20240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:42.774102 20240 hybrid_clock.cc:648] HybridClock initialized: now 1781888142774058 us; error 46 us; skew 500 ppm
I20260619 16:55:42.774459 20240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:42 dist-test-slave-3ffd krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:42.778024 20240 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:55:42.779682 20240 webserver.cc:460] Webserver started at http://127.16.48.1:44529/ using document root <none> and password file <none>
I20260619 16:55:42.780560 20240 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:42.780727 20240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:42.781062 20240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:42.784332 20240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "843bce983db84e7a803e264a93b4b33c"
format_stamp: "Formatted at 2026-06-19 16:55:42 on dist-test-slave-3ffd"
server_key: "97441749e575693953073fa3f6036dc4"
server_key_iv: "45b0eb0001e5caa38790c0579722f5ed"
server_key_version: "encryptionkey@0"
I20260619 16:55:42.785254 20240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "843bce983db84e7a803e264a93b4b33c"
format_stamp: "Formatted at 2026-06-19 16:55:42 on dist-test-slave-3ffd"
server_key: "97441749e575693953073fa3f6036dc4"
server_key_iv: "45b0eb0001e5caa38790c0579722f5ed"
server_key_version: "encryptionkey@0"
I20260619 16:55:42.791002 20240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:55:42.794970 20255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:42.796820 20240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:55:42.797070 20240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "843bce983db84e7a803e264a93b4b33c"
format_stamp: "Formatted at 2026-06-19 16:55:42 on dist-test-slave-3ffd"
server_key: "97441749e575693953073fa3f6036dc4"
server_key_iv: "45b0eb0001e5caa38790c0579722f5ed"
server_key_version: "encryptionkey@0"
I20260619 16:55:42.797333 20240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:42.815699 20240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:42.820509 20240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:42.820860 20240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:42.821951 20240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:42.823596 20240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:42.823721 20240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:42.823882 20240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:42.823952 20240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:42.858896 20240 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:35351
I20260619 16:55:42.858999 20367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:35351 every 8 connection(s)
I20260619 16:55:42.860592 20240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:42.867093 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 20240
I20260619 16:55:42.867353 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:42.868279 16576 external_mini_cluster.cc:1428] Setting key bd6e3d63cf5f4313792d1589dc2947ee
Jun 19 16:55:42 dist-test-slave-3ffd krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:42.887149 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:42.863737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:50977 (local address 127.16.48.62:37141)
0619 16:55:42.864299 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:42.864304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:42.865399 (+ 1095us) server_negotiation.cc:407] Connection header received
0619 16:55:42.867620 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:42.867628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:42.867752 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:42.867886 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:42.870021 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:42.870989 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:42.874106 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:42.874550 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:42.874677 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:42.879942 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:42.879949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:42.879952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:42.879993 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:42.883434 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:42.884487 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:42.884493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:42.884497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:42.884638 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:42.885138 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:42.885144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:42.885147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:42.885469 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:42.885636 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:42.886519 (+ 883us) server_negotiation.cc:299] Negotiation successful
0619 16:55:42.886771 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":284,"threads_started":1}
I20260619 16:55:42.888321 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:42.864307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37141 (local address 127.16.48.1:50977)
0619 16:55:42.865180 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:42.865246 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:42.867164 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:42.868254 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:42.868265 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:42.868933 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:42.869803 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:42.869827 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:42.871228 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:42.871233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:42.873852 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:42.873884 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:42.874833 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:42.874843 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:42.875245 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:42.876318 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:42.876356 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:42.879582 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:42.883701 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:42.883715 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:42.883740 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:42.884330 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:42.884773 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:42.884778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:42.884782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:42.885003 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:42.885621 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:42.885631 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:42.886093 (+ 462us) client_negotiation.cc:769] Sending connection context
0619 16:55:42.886396 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 16:55:42.887024 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":366,"threads_started":1}
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 16:55:42.890365 20368 heartbeater.cc:347] Connected to a master server at 127.16.48.62:37141
I20260619 16:55:42.890735 20368 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:42.891598 20368 heartbeater.cc:511] Master 127.16.48.62:37141 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:55:42.894385 20179 ts_manager.cc:194] Registered new tserver with Master: 843bce983db84e7a803e264a93b4b33c (127.16.48.1:35351)
I20260619 16:55:42.896083 20179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:50977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:55:42.919870 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37141
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:43.286479 20374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:43.286799 20374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:43.286986 20374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:43.294823 20374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:43.294970 20374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:43.295217 20374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:43.306025 20374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:43.307477 20374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:43.309118 20374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:43.323302 20379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:43.323262 20380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:43.323325 20382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:43.324049 20374 server_base.cc:1034] running on GCE node
I20260619 16:55:43.324832 20374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:43.325906 20374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:43.327121 20374 hybrid_clock.cc:648] HybridClock initialized: now 1781888143327043 us; error 71 us; skew 500 ppm
I20260619 16:55:43.327514 20374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:43 dist-test-slave-3ffd krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888143, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:43.331172 20374 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:55:43.333138 20374 webserver.cc:460] Webserver started at http://127.16.48.2:38113/ using document root <none> and password file <none>
I20260619 16:55:43.333938 20374 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:43.334076 20374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:43.334416 20374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:43.337893 20374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "ba2a59a8e02547d19c304cf6839ad49e"
format_stamp: "Formatted at 2026-06-19 16:55:43 on dist-test-slave-3ffd"
server_key: "2955a0de7f61a603954f7a33fb4c0300"
server_key_iv: "f523245360be901bc3b845653b6fc03f"
server_key_version: "encryptionkey@0"
I20260619 16:55:43.338792 20374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "ba2a59a8e02547d19c304cf6839ad49e"
format_stamp: "Formatted at 2026-06-19 16:55:43 on dist-test-slave-3ffd"
server_key: "2955a0de7f61a603954f7a33fb4c0300"
server_key_iv: "f523245360be901bc3b845653b6fc03f"
server_key_version: "encryptionkey@0"
I20260619 16:55:43.344821 20374 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 16:55:43.348791 20389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:43.350335 20374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:43.350529 20374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "ba2a59a8e02547d19c304cf6839ad49e"
format_stamp: "Formatted at 2026-06-19 16:55:43 on dist-test-slave-3ffd"
server_key: "2955a0de7f61a603954f7a33fb4c0300"
server_key_iv: "f523245360be901bc3b845653b6fc03f"
server_key_version: "encryptionkey@0"
I20260619 16:55:43.350754 20374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:43.369498 20374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:43.374519 20374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:43.374846 20374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:43.375869 20374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:43.377348 20374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:43.377449 20374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:43.377600 20374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:43.377669 20374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:43.412222 20374 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:43579
I20260619 16:55:43.412278 20501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:43579 every 8 connection(s)
I20260619 16:55:43.413638 20374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:43.421419 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 20374
I20260619 16:55:43.421628 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:43.422060 16576 external_mini_cluster.cc:1428] Setting key 037f8af4554b8c29bf655019d166292a
Jun 19 16:55:43 dist-test-slave-3ffd krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888143, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:43.435649 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:43.416336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:33653 (local address 127.16.48.62:37141)
0619 16:55:43.416811 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:43.416815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:43.417961 (+ 1146us) server_negotiation.cc:407] Connection header received
0619 16:55:43.419986 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:43.419992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:43.420098 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:43.420256 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:43.422088 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:43.423021 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:43.424946 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:43.425430 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:43.425575 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:43.429471 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:43.429477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:43.429480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:43.429518 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:43.432311 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:43.433190 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:43.433196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:43.433198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:43.433332 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:43.433821 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:43.433825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:43.433827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:43.434105 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:43.434264 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:43.435017 (+ 753us) server_negotiation.cc:299] Negotiation successful
0619 16:55:43.435255 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260619 16:55:43.436923 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:43.417021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37141 (local address 127.16.48.2:33653)
0619 16:55:43.417781 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:43.417824 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:43.419635 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:43.420549 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:43.420560 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:43.421136 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:43.421889 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:43.421911 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:43.423226 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:43.423230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:43.424785 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:43.424800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:43.425691 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:43.425699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:43.425960 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:43.426729 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:43.426759 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:43.429161 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:43.432500 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:43.432513 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:43.432535 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:43.433030 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:43.433454 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:43.433459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:43.433463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:43.433686 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:43.434242 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:43.434251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:43.434644 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 16:55:43.434932 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 16:55:43.435539 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":295,"threads_started":1}
I20260619 16:55:43.438781 20502 heartbeater.cc:347] Connected to a master server at 127.16.48.62:37141
I20260619 16:55:43.439129 20502 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:43.439930 20502 heartbeater.cc:511] Master 127.16.48.62:37141 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
I20260619 16:55:43.441850 20179 ts_manager.cc:194] Registered new tserver with Master: ba2a59a8e02547d19c304cf6839ad49e (127.16.48.2:43579)
I20260619 16:55:43.442672 20179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:33653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:55:43.471782 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37141
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:43.819414 20508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:43.819746 20508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:43.819844 20508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:43.827610 20508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:43.827749 20508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:43.827872 20508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:43.838191 20508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:43.839565 20508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:43.840981 20508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:43.855645 20513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:43.855558 20514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:43.855618 20516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:43.856606 20508 server_base.cc:1034] running on GCE node
I20260619 16:55:43.857240 20508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:43.858304 20508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:43.859526 20508 hybrid_clock.cc:648] HybridClock initialized: now 1781888143859476 us; error 51 us; skew 500 ppm
I20260619 16:55:43.859869 20508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:43 dist-test-slave-3ffd krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888143, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:43.863272 20508 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:55:43.864905 20508 webserver.cc:460] Webserver started at http://127.16.48.3:37725/ using document root <none> and password file <none>
I20260619 16:55:43.865665 20508 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:43.865816 20508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:43.866144 20508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:43.869395 20508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "852b9a71cbdd45ac8a5bb088b221dca5"
format_stamp: "Formatted at 2026-06-19 16:55:43 on dist-test-slave-3ffd"
server_key: "f1bb962326321296ff3277b5de820ef6"
server_key_iv: "013beca34f26ea0b456e00136cfae7e3"
server_key_version: "encryptionkey@0"
I20260619 16:55:43.870350 20508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "852b9a71cbdd45ac8a5bb088b221dca5"
format_stamp: "Formatted at 2026-06-19 16:55:43 on dist-test-slave-3ffd"
server_key: "f1bb962326321296ff3277b5de820ef6"
server_key_iv: "013beca34f26ea0b456e00136cfae7e3"
server_key_version: "encryptionkey@0"
I20260619 16:55:43.876446 20508 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 16:55:43.880398 20523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:43.882062 20508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:43.882275 20508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "852b9a71cbdd45ac8a5bb088b221dca5"
format_stamp: "Formatted at 2026-06-19 16:55:43 on dist-test-slave-3ffd"
server_key: "f1bb962326321296ff3277b5de820ef6"
server_key_iv: "013beca34f26ea0b456e00136cfae7e3"
server_key_version: "encryptionkey@0"
I20260619 16:55:43.882507 20508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:43.900792 20368 heartbeater.cc:503] Master 127.16.48.62:37141 was elected leader, sending a full tablet report...
I20260619 16:55:43.906515 20508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:43.910933 20508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:43.911259 20508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:43.912362 20508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:43.913779 20508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:43.913882 20508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:43.914031 20508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:43.914108 20508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:43.949002 20508 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:35067
I20260619 16:55:43.949065 20635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:35067 every 8 connection(s)
I20260619 16:55:43.950441 20508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:43.952731 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 20508
I20260619 16:55:43.952941 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:43.953497 16576 external_mini_cluster.cc:1428] Setting key db91bc090c1838bcd5185d9ff4a824dc
Jun 19 16:55:43 dist-test-slave-3ffd krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888143, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:43.973266 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:43.953642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:44511 (local address 127.16.48.62:37141)
0619 16:55:43.954162 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:43.954170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:43.955502 (+ 1332us) server_negotiation.cc:407] Connection header received
0619 16:55:43.957857 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:43.957861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:43.957928 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:43.958023 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:43.959664 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:43.960595 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:43.962205 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:43.962618 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:43.962720 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:43.966452 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:43.966458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:43.966461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:43.966498 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:43.969882 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:43.970759 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:43.970764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:43.970766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:43.970907 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:43.971463 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:43.971467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:43.971469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:43.971788 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:43.971919 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:43.972755 (+ 836us) server_negotiation.cc:299] Negotiation successful
0619 16:55:43.972942 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":256,"threads_started":1}
I20260619 16:55:43.974401 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:43.954373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37141 (local address 127.16.48.3:44511)
0619 16:55:43.955321 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:43.955379 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:43.957583 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:43.958256 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:43.958265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:43.958859 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:43.959469 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:43.959487 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:43.960751 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:43.960755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:43.962050 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:43.962061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:43.962825 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:43.962830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:43.963048 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:43.963774 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:43.963802 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:43.966181 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:43.970034 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:43.970044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:43.970061 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:43.970626 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:43.971038 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:43.971042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:43.971044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:43.971252 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:43.971919 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:43.971927 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:43.972365 (+ 438us) client_negotiation.cc:769] Sending connection context
0619 16:55:43.972667 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 16:55:43.973160 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":345,"threads_started":1}
I20260619 16:55:43.976164 20636 heartbeater.cc:347] Connected to a master server at 127.16.48.62:37141
I20260619 16:55:43.976506 20636 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:43.977288 20636 heartbeater.cc:511] Master 127.16.48.62:37141 requested a full tablet report, sending...
I20260619 16:55:43.978852 20179 ts_manager.cc:194] Registered new tserver with Master: 852b9a71cbdd45ac8a5bb088b221dca5 (127.16.48.3:35067)
I20260619 16:55:43.979713 20179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:44511
I20260619 16:55:43.982398 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:55:44 dist-test-slave-3ffd krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:55:44 dist-test-slave-3ffd krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:44.033895 20644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:44.018302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.16.48.1:35351)
0619 16:55:44.018978 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:44.018985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:44.019013 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:55:44.019074 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:44.019080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:44.019307 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:44.019444 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:44.020950 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.022016 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.023260 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.023846 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.023951 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:44.028201 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:44.028218 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:44.028237 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:44.028277 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:44.031014 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.031759 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.031767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.031770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.031900 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.032336 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.032340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.032342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.032714 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:44.032955 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:44.033278 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 16:55:44.033489 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":372,"threads_started":1}
Jun 19 16:55:44 dist-test-slave-3ffd krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:44.052884 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:44.038840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.16.48.62:37141)
0619 16:55:44.039068 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:44.039073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:44.039173 (+ 100us) server_negotiation.cc:407] Connection header received
0619 16:55:44.039316 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:44.039319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:44.039383 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:44.039468 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:44.040813 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.041559 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.042844 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.043250 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.043351 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:44.047453 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:44.047458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:44.047462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:44.047499 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:44.050344 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.051098 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.051102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.051104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.051221 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.051531 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.051534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.051536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.051853 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:44.052036 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:44.052378 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:55:44.052549 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 16:55:44.053488 20179 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50996
I20260619 16:55:44.054561 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 20240
I20260619 16:55:44.067495 20644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:44.055545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.16.48.1:35351)
0619 16:55:44.055764 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:44.055771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:44.055861 (+ 90us) server_negotiation.cc:407] Connection header received
0619 16:55:44.056013 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:44.056017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:44.056079 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:44.056236 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:44.057402 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.058072 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.059321 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.059766 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.059875 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:44.062287 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:44.062292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:44.062295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:44.062326 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:44.064957 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.065704 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.065709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.065711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.065837 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.066189 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.066193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.066196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.066490 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:44.066700 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:44.066976 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 16:55:44.067159 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:55:44.068269 20363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:44.151603 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 20240
I20260619 16:55:44.168336 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 20374
Jun 19 16:55:44 dist-test-slave-3ffd krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:55:44.185931 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:44.169275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.16.48.2:43579)
0619 16:55:44.169898 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:44.169905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:44.169933 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:55:44.170003 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:44.170010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:44.170233 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:44.170373 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:44.171810 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.173226 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.174608 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.175254 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.175442 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:44.179793 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:44.179810 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:44.179826 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:44.179873 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:44.182903 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.183742 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.183750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.183753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.183891 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.184385 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.184389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.184391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.184763 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:44.185004 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:44.185281 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 16:55:44.185492 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":311,"threads_started":1}
I20260619 16:55:44.187235 20497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:44.271171 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 20374
I20260619 16:55:44.287346 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 20508
Jun 19 16:55:44 dist-test-slave-3ffd krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:55:44.304401 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:44.288426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.16.48.3:35067)
0619 16:55:44.289167 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:44.289174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:44.289204 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:55:44.289266 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:44.289273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:44.289568 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:44.289708 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:44.291055 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.292228 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.293493 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:44.294074 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:44.294184 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:44.298571 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:44.298586 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:44.298603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:44.298646 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:44.301581 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.302346 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.302361 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.302363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.302503 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:44.302863 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:44.302867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:44.302869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:44.303235 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:44.303467 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:44.303712 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 16:55:44.303919 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":401,"threads_started":1}
I20260619 16:55:44.305794 20631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:44.389427 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 20508
I20260619 16:55:44.405310 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 20147
I20260619 16:55:44.406549 20209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:44.509611 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 20147
2026-06-19T16:55:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4062 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:44 dist-test-slave-3ffd krb5kdc[20656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:44 dist-test-slave-3ffd krb5kdc[20656](info): set up 2 sockets
Jun 19 16:55:44 dist-test-slave-3ffd krb5kdc[20656](info): commencing operation
krb5kdc: starting...
W20260619 16:55:45.608868 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:45 dist-test-slave-3ffd krb5kdc[20656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:45.823344 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:44691
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:44691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:46.171907 20669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:46.172369 20669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:46.172557 20669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:46.180207 20669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:46.180354 20669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:46.180456 20669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:46.180552 20669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:46.180624 20669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:46.191123 20669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:44691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:44691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:46.192658 20669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:46.194087 20669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:46.206322 20675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:46.206353 20674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:46.206423 20677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:46.207358 20669 server_base.cc:1034] running on GCE node
I20260619 16:55:46.208251 20669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:46.209780 20669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:46.211028 20669 hybrid_clock.cc:648] HybridClock initialized: now 1781888146210956 us; error 67 us; skew 500 ppm
I20260619 16:55:46.211472 20669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:46 dist-test-slave-3ffd krb5kdc[20656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:46.220003 20669 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:46.221312 20669 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:46.221444 20669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:46.221828 20669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:46.225337 20669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "18168eedfdbf4c0592d4d4115fb040b7"
format_stamp: "Formatted at 2026-06-19 16:55:46 on dist-test-slave-3ffd"
server_key: "044037026e4fa553a653507a6d7c0490"
server_key_iv: "a350379e973a76cee0ec121f27c416a7"
server_key_version: "encryptionkey@0"
I20260619 16:55:46.226264 20669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "18168eedfdbf4c0592d4d4115fb040b7"
format_stamp: "Formatted at 2026-06-19 16:55:46 on dist-test-slave-3ffd"
server_key: "044037026e4fa553a653507a6d7c0490"
server_key_iv: "a350379e973a76cee0ec121f27c416a7"
server_key_version: "encryptionkey@0"
I20260619 16:55:46.231631 20669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:55:46.235466 20683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:46.236969 20669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:46.237171 20669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "18168eedfdbf4c0592d4d4115fb040b7"
format_stamp: "Formatted at 2026-06-19 16:55:46 on dist-test-slave-3ffd"
server_key: "044037026e4fa553a653507a6d7c0490"
server_key_iv: "a350379e973a76cee0ec121f27c416a7"
server_key_version: "encryptionkey@0"
I20260619 16:55:46.237443 20669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:46.250733 20669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:46.255364 20669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:46.255687 20669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:46.274129 20669 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:44691
I20260619 16:55:46.274173 20734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:44691 every 8 connection(s)
I20260619 16:55:46.275162 20669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:46.279268 20735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:46.284734 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 20669
I20260619 16:55:46.284997 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:46.285791 16576 external_mini_cluster.cc:1428] Setting key 2e6a1d2844658f798c797a5047562eba
I20260619 16:55:46.292256 20735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7: Bootstrap starting.
I20260619 16:55:46.296406 20735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:46.297778 20735 log.cc:824] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:46.301002 20735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7: No bootstrap required, opened a new log
Jun 19 16:55:46 dist-test-slave-3ffd krb5kdc[20656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:46.312436 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:46.288205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50056 (local address 127.16.48.62:44691)
0619 16:55:46.289129 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:46.289145 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:46.289196 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:55:46.291038 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:46.291067 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:46.291549 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:46.291918 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:46.293941 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:46.295396 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:46.297341 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:46.298184 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:46.298410 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:46.302664 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:46.302676 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:46.302700 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:46.302763 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:46.307246 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:46.308169 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:46.308180 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:46.308192 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:46.308431 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:46.308837 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:46.308842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:46.308845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:46.309660 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:46.309975 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:46.310379 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 16:55:46.310891 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":430,"threads_started":1}
I20260619 16:55:46.312254 20735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18168eedfdbf4c0592d4d4115fb040b7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44691 } }
I20260619 16:55:46.312738 20735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:46.312844 20735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18168eedfdbf4c0592d4d4115fb040b7, State: Initialized, Role: FOLLOWER
I20260619 16:55:46.313527 20735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18168eedfdbf4c0592d4d4115fb040b7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44691 } }
I20260619 16:55:46.313823 20735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:46.313968 20735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:46.314194 20735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:46.316881 20735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18168eedfdbf4c0592d4d4115fb040b7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44691 } }
I20260619 16:55:46.317404 20735 leader_election.cc:302] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18168eedfdbf4c0592d4d4115fb040b7; no voters:
I20260619 16:55:46.318009 20735 leader_election.cc:288] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:46.318166 20740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:46.318931 20740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [term 1 LEADER]: Becoming Leader. State: Replica: 18168eedfdbf4c0592d4d4115fb040b7, State: Running, Role: LEADER
I20260619 16:55:46.319552 20740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18168eedfdbf4c0592d4d4115fb040b7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44691 } }
I20260619 16:55:46.320222 20735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:46.322382 20741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18168eedfdbf4c0592d4d4115fb040b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18168eedfdbf4c0592d4d4115fb040b7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44691 } } }
I20260619 16:55:46.322772 20741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:46.323305 20742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18168eedfdbf4c0592d4d4115fb040b7. Latest consensus state: current_term: 1 leader_uuid: "18168eedfdbf4c0592d4d4115fb040b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18168eedfdbf4c0592d4d4115fb040b7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44691 } } }
I20260619 16:55:46.323588 20742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:46.325552 20745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:46.330958 20745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:46.341323 20745 catalog_manager.cc:1347] Generated new cluster ID: 1c59d5faf8c34e9bb2755c14ab85ed4e
I20260619 16:55:46.341521 20745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:46.361310 20745 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:46.362949 20745 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:46.371735 20745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7: Generated new TSK 0
I20260619 16:55:46.372819 20745 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:55:46.434505 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44691
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:46.789767 20761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:46.790140 20761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:46.790321 20761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:46.797988 20761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:46.798177 20761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:46.798380 20761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:46.809149 20761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:46.810562 20761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:46.812101 20761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:46.826359 20767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:46.826746 20766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:46.826833 20769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:46.827630 20761 server_base.cc:1034] running on GCE node
I20260619 16:55:46.828346 20761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:46.829370 20761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:46.830592 20761 hybrid_clock.cc:648] HybridClock initialized: now 1781888146830535 us; error 54 us; skew 500 ppm
I20260619 16:55:46.830945 20761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:46 dist-test-slave-3ffd krb5kdc[20656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:46.834439 20761 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:55:46.835777 20761 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:46.835883 20761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:46.836342 20761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:46.839550 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "9afaa552bb0e4828a63b2ab4a82f77d1"
format_stamp: "Formatted at 2026-06-19 16:55:46 on dist-test-slave-3ffd"
server_key: "72c8c1be47e8693fc1c2a85071f421df"
server_key_iv: "a04a4f99c004039d5ef8021fd1abad12"
server_key_version: "encryptionkey@0"
I20260619 16:55:46.840560 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "9afaa552bb0e4828a63b2ab4a82f77d1"
format_stamp: "Formatted at 2026-06-19 16:55:46 on dist-test-slave-3ffd"
server_key: "72c8c1be47e8693fc1c2a85071f421df"
server_key_iv: "a04a4f99c004039d5ef8021fd1abad12"
server_key_version: "encryptionkey@0"
I20260619 16:55:46.846539 20761 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:55:46.850594 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:46.852368 20761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:46.852558 20761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "9afaa552bb0e4828a63b2ab4a82f77d1"
format_stamp: "Formatted at 2026-06-19 16:55:46 on dist-test-slave-3ffd"
server_key: "72c8c1be47e8693fc1c2a85071f421df"
server_key_iv: "a04a4f99c004039d5ef8021fd1abad12"
server_key_version: "encryptionkey@0"
I20260619 16:55:46.852802 20761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:46.875013 20761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:46.879732 20761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:46.880031 20761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:46.881067 20761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:46.882475 20761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:46.882567 20761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:46.882680 20761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:46.882757 20761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:46.916666 20761 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:33283
I20260619 16:55:46.916733 20887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:33283 every 8 connection(s)
I20260619 16:55:46.917609 20761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:46.918154 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 20761
I20260619 16:55:46.918395 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:46.919067 16576 external_mini_cluster.cc:1428] Setting key 58e2eb946dc24315ebe8827a5bde0bf5
Jun 19 16:55:46 dist-test-slave-3ffd krb5kdc[20656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:55:46.948168 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:46.921221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:53817 (local address 127.16.48.62:44691)
0619 16:55:46.930487 (+ 9266us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:46.930494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:46.930524 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:55:46.931827 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:46.931833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:46.931934 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:46.932072 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:46.933905 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:46.934677 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:46.936644 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:46.937081 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:46.937252 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:46.941240 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:46.941245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:46.941248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:46.941285 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:46.944253 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:46.945385 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:46.945391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:46.945394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:46.945566 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:46.946037 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:46.946046 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:46.946052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:46.946474 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:46.946637 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:46.947394 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 16:55:46.947661 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9114,"thread_start_us":8980,"threads_started":1}
I20260619 16:55:46.949085 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:46.922031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44691 (local address 127.16.48.1:53817)
0619 16:55:46.929055 (+ 7024us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:46.929151 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:46.931436 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:46.932401 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:46.932413 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:46.933016 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:46.933686 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:46.933707 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:46.934888 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:46.934892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:46.936450 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:46.936472 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:46.937371 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:46.937378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:46.937651 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:46.938455 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:46.938483 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:46.940943 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:46.944427 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:46.944436 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:46.944452 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:46.945188 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:46.945710 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:46.945714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:46.945716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:46.945904 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:46.946604 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:46.946611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:46.946976 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 16:55:46.947216 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 16:55:46.947677 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6546,"thread_start_us":6260,"threads_started":1}
I20260619 16:55:46.951170 20888 heartbeater.cc:347] Connected to a master server at 127.16.48.62:44691
I20260619 16:55:46.951551 20888 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:46.952440 20888 heartbeater.cc:511] Master 127.16.48.62:44691 requested a full tablet report, sending...
I20260619 16:55:46.954838 20700 ts_manager.cc:194] Registered new tserver with Master: 9afaa552bb0e4828a63b2ab4a82f77d1 (127.16.48.1:33283)
I20260619 16:55:46.956559 20700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:53817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:55:46.971503 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44691
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:47.318373 20894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:47.318759 20894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:47.318933 20894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:47.326804 20894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:47.326972 20894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:47.327155 20894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:47.337249 20894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:47.338714 20894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:47.340167 20894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:47.353739 20899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:47.353716 20900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:47.353744 20902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:47.355170 20894 server_base.cc:1034] running on GCE node
I20260619 16:55:47.355831 20894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:47.357034 20894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:47.358238 20894 hybrid_clock.cc:648] HybridClock initialized: now 1781888147358182 us; error 41 us; skew 500 ppm
I20260619 16:55:47.358578 20894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:47 dist-test-slave-3ffd krb5kdc[20656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:47.362098 20894 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:55:47.363349 20894 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:47.363488 20894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:47.363824 20894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:47.367172 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "59b1657361fe4eb18d7ad52a1da75b3c"
format_stamp: "Formatted at 2026-06-19 16:55:47 on dist-test-slave-3ffd"
server_key: "985dd2905462284c4eb4c7fc4a2ebc64"
server_key_iv: "6a8041af1b4cbb12691cb7336cbf912c"
server_key_version: "encryptionkey@0"
I20260619 16:55:47.368175 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "59b1657361fe4eb18d7ad52a1da75b3c"
format_stamp: "Formatted at 2026-06-19 16:55:47 on dist-test-slave-3ffd"
server_key: "985dd2905462284c4eb4c7fc4a2ebc64"
server_key_iv: "6a8041af1b4cbb12691cb7336cbf912c"
server_key_version: "encryptionkey@0"
I20260619 16:55:47.374169 20894 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:55:47.378244 20908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:47.379819 20894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:55:47.380034 20894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "59b1657361fe4eb18d7ad52a1da75b3c"
format_stamp: "Formatted at 2026-06-19 16:55:47 on dist-test-slave-3ffd"
server_key: "985dd2905462284c4eb4c7fc4a2ebc64"
server_key_iv: "6a8041af1b4cbb12691cb7336cbf912c"
server_key_version: "encryptionkey@0"
I20260619 16:55:47.380357 20894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:47.393568 20894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:47.398168 20894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:47.398473 20894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:47.399525 20894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:47.400991 20894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:47.401117 20894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:47.401247 20894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:47.401331 20894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:47.436298 20894 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:36365
I20260619 16:55:47.436359 21020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:36365 every 8 connection(s)
I20260619 16:55:47.437283 20894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:47.442639 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 20894
I20260619 16:55:47.442850 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:47.443279 16576 external_mini_cluster.cc:1428] Setting key b277f8ba7e480266649eedd66004964e
Jun 19 16:55:47 dist-test-slave-3ffd krb5kdc[20656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:47.461355 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:47.440166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:56821 (local address 127.16.48.62:44691)
0619 16:55:47.440332 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:47.440337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:47.441832 (+ 1495us) server_negotiation.cc:407] Connection header received
0619 16:55:47.444097 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:47.444103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:47.444220 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:47.444338 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:47.446372 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:47.447087 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:47.448883 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:47.449325 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:47.449471 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:47.453434 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:47.453444 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:47.453452 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:47.453502 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:47.457329 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:47.458332 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:47.458342 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:47.458349 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:47.458528 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:47.458987 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:47.458995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:47.459000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:47.459422 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:47.459583 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:47.460570 (+ 987us) server_negotiation.cc:299] Negotiation successful
0619 16:55:47.460865 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:55:47.462141 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:47.440749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44691 (local address 127.16.48.2:56821)
0619 16:55:47.441603 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:47.441694 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:47.443701 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:47.444682 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:47.444695 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:47.445530 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:47.446178 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:47.446198 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:47.447272 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:47.447278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:47.448713 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:47.448727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:47.449585 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:47.449592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:47.449947 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:47.450747 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:47.450775 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:47.453145 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:47.457513 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:47.457522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:47.457539 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:47.458157 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:47.458654 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:47.458658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:47.458661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:47.458857 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:47.459583 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:47.459590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:47.460071 (+ 481us) client_negotiation.cc:769] Sending connection context
0619 16:55:47.460419 (+ 348us) client_negotiation.cc:240] Negotiation successful
0619 16:55:47.460852 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":299,"threads_started":1}
I20260619 16:55:47.463765 21021 heartbeater.cc:347] Connected to a master server at 127.16.48.62:44691
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
I20260619 16:55:47.464169 21021 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:47.464883 21021 heartbeater.cc:511] Master 127.16.48.62:44691 requested a full tablet report, sending...
I20260619 16:55:47.466554 20700 ts_manager.cc:194] Registered new tserver with Master: 59b1657361fe4eb18d7ad52a1da75b3c (127.16.48.2:36365)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
I20260619 16:55:47.467629 20700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:56821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:55:47.493235 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44691
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:47.840994 21026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:47.841328 21026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:47.841431 21026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:47.849303 21026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:47.849445 21026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:47.849567 21026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:47.859987 21026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:47.861356 21026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:47.862675 21026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:47.876799 21031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:47.876765 21034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:47.876844 21032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:47.877594 21026 server_base.cc:1034] running on GCE node
I20260619 16:55:47.878309 21026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:47.879383 21026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:47.880589 21026 hybrid_clock.cc:648] HybridClock initialized: now 1781888147880540 us; error 52 us; skew 500 ppm
I20260619 16:55:47.880949 21026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:47 dist-test-slave-3ffd krb5kdc[20656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:47.884509 21026 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:55:47.885866 21026 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:47.886004 21026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:47.886338 21026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:47.889626 21026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "df6edd78e22746aebd3d9c799b5f1743"
format_stamp: "Formatted at 2026-06-19 16:55:47 on dist-test-slave-3ffd"
server_key: "150e053cb8f0d5ea1abd5ef2c355c3dd"
server_key_iv: "7577ab6e3a2afeeede193736c4091f36"
server_key_version: "encryptionkey@0"
I20260619 16:55:47.890547 21026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "df6edd78e22746aebd3d9c799b5f1743"
format_stamp: "Formatted at 2026-06-19 16:55:47 on dist-test-slave-3ffd"
server_key: "150e053cb8f0d5ea1abd5ef2c355c3dd"
server_key_iv: "7577ab6e3a2afeeede193736c4091f36"
server_key_version: "encryptionkey@0"
I20260619 16:55:47.896503 21026 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 16:55:47.900981 21040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:47.903096 21026 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:55:47.903295 21026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "df6edd78e22746aebd3d9c799b5f1743"
format_stamp: "Formatted at 2026-06-19 16:55:47 on dist-test-slave-3ffd"
server_key: "150e053cb8f0d5ea1abd5ef2c355c3dd"
server_key_iv: "7577ab6e3a2afeeede193736c4091f36"
server_key_version: "encryptionkey@0"
I20260619 16:55:47.903533 21026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:47.920397 21026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:47.925526 21026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:47.925832 21026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:47.926949 21026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:47.928411 21026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:47.928534 21026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:47.928671 21026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:47.928762 21026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:47.960963 20888 heartbeater.cc:503] Master 127.16.48.62:44691 was elected leader, sending a full tablet report...
I20260619 16:55:47.963572 21026 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:41083
I20260619 16:55:47.963615 21152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:41083 every 8 connection(s)
I20260619 16:55:47.964529 21026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:47.974279 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 21026
I20260619 16:55:47.974509 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:47.974992 16576 external_mini_cluster.cc:1428] Setting key 3f242f1692daffc0309774d8e97fe9f7
Jun 19 16:55:47 dist-test-slave-3ffd krb5kdc[20656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:47.985715 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:47.967064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:49067 (local address 127.16.48.62:44691)
0619 16:55:47.967536 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:47.967540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:47.968488 (+ 948us) server_negotiation.cc:407] Connection header received
0619 16:55:47.970384 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:47.970390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:47.970485 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:47.970605 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:47.972391 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:47.973178 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:47.975024 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:47.975582 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:47.975750 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:47.979881 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:47.979886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:47.979889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:47.979923 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:47.982412 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:47.983244 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:47.983248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:47.983251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:47.983380 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:47.983789 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:47.983792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:47.983794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:47.984077 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:47.984277 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:47.985078 (+ 801us) server_negotiation.cc:299] Negotiation successful
0619 16:55:47.985281 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":264,"threads_started":1}
I20260619 16:55:47.986652 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:47.967570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44691 (local address 127.16.48.3:49067)
0619 16:55:47.968330 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:47.968371 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:47.970049 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:47.970876 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:47.970886 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:47.971476 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:47.972104 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:47.972195 (+ 91us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:47.973369 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:47.973374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:47.974853 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:47.974868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:47.975883 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:47.975893 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:47.976285 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:47.977161 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:47.977207 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:47.979663 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:47.982576 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:47.982585 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:47.982600 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:47.983089 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:47.983487 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:47.983491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:47.983493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:47.983686 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:47.984253 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:47.984259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:47.984718 (+ 459us) client_negotiation.cc:769] Sending connection context
0619 16:55:47.984980 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 16:55:47.985424 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":272,"threads_started":1}
I20260619 16:55:47.988255 21153 heartbeater.cc:347] Connected to a master server at 127.16.48.62:44691
I20260619 16:55:47.988590 21153 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:47.989482 21153 heartbeater.cc:511] Master 127.16.48.62:44691 requested a full tablet report, sending...
I20260619 16:55:47.991117 20700 ts_manager.cc:194] Registered new tserver with Master: df6edd78e22746aebd3d9c799b5f1743 (127.16.48.3:41083)
I20260619 16:55:47.991976 20700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:49067
I20260619 16:55:48.003700 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:55:48.022166 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.008744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.16.48.62:44691)
0619 16:55:48.008974 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.008983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.009407 (+ 424us) server_negotiation.cc:407] Connection header received
0619 16:55:48.009602 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.009606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.009676 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.009768 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:48.011182 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.011914 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.013326 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.013783 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.013897 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.016324 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:48.016329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:48.016333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:48.016369 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:48.019104 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:48.019973 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:48.019986 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:48.019991 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:48.020175 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:48.020605 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:48.020610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:48.020612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:48.020991 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:48.021240 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.021619 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.021811 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:55:48.026681 20700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:55:48.033685 20700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:48.060787 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.048574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.16.48.1:33283)
0619 16:55:48.050238 (+ 1664us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.050247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.050772 (+ 525us) server_negotiation.cc:407] Connection header received
0619 16:55:48.051072 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.051082 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.051382 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.051579 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:48.053310 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.054772 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.057977 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.059410 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.059560 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.059816 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.060282 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.060426 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1491,"thread_start_us":1345,"threads_started":1}
I20260619 16:55:48.061269 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.050753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:36365 (local address 127.0.0.1:60216)
0619 16:55:48.052466 (+ 1713us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.052485 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:48.052600 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:48.053250 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:48.053255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:48.053279 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:48.053604 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.053612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.054880 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.054888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.057396 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.057414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.060263 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.060270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.060444 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.060474 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:55:48.060576 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:55:48.060814 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1583,"mutex_wait_us":63,"thread_start_us":1426,"threads_started":1}
I20260619 16:55:48.061645 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.046948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:41083 (local address 127.0.0.1:43336)
0619 16:55:48.048812 (+ 1864us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.048864 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:48.049094 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:48.051930 (+ 2836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:48.051933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:48.051949 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:48.052449 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.052466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.055879 (+ 3413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.055885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.058257 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.058271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.060701 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.060707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.060864 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.060892 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:55:48.060968 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:55:48.061093 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1505,"mutex_wait_us":3,"thread_start_us":3417,"threads_started":1}
I20260619 16:55:48.061995 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.047709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:33283 (local address 127.0.0.1:59350)
0619 16:55:48.050665 (+ 2956us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.050682 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:48.050876 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:48.051900 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:48.051910 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:48.051941 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:48.053083 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.053096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.054959 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.054963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.057807 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.057824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.059633 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.059641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.059888 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.059950 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 16:55:48.060102 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 16:55:48.061582 (+ 1480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2802,"mutex_wait_us":878,"thread_start_us":1068,"threads_started":1}
I20260619 16:55:48.062039 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.051004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.16.48.2:36365)
0619 16:55:48.052769 (+ 1765us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.052779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.052812 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:55:48.052888 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.052893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.053070 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.053262 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:48.053770 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.054734 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.057748 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.059410 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.059560 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.059784 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.060649 (+ 865us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.060848 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1622,"thread_start_us":1525,"threads_started":1}
I20260619 16:55:48.062260 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.047278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.16.48.3:41083)
0619 16:55:48.049803 (+ 2525us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.049814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.049852 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:55:48.049971 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.049978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.051739 (+ 1761us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.052675 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:48.052986 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.055708 (+ 2722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.058472 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.060470 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.061361 (+ 891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.061634 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.061745 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.061916 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2353,"thread_start_us":2185,"threads_started":1}
I20260619 16:55:48.065742 21088 tablet_service.cc:1467] Processing CreateTablet for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55 (DEFAULT_TABLE table=test-table [id=cba558b678de47d19e90d768281a73ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:48.065840 20823 tablet_service.cc:1467] Processing CreateTablet for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55 (DEFAULT_TABLE table=test-table [id=cba558b678de47d19e90d768281a73ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:48.065837 20956 tablet_service.cc:1467] Processing CreateTablet for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55 (DEFAULT_TABLE table=test-table [id=cba558b678de47d19e90d768281a73ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:48.066979 21088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:48.067200 20956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:48.067215 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:48.080330 21172 tablet_bootstrap.cc:492] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: Bootstrap starting.
I20260619 16:55:48.082568 21173 tablet_bootstrap.cc:492] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: Bootstrap starting.
I20260619 16:55:48.084419 21172 tablet_bootstrap.cc:654] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:48.085964 21172 log.cc:824] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:48.086422 21173 tablet_bootstrap.cc:654] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:48.088073 21173 log.cc:824] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:48.089761 21174 tablet_bootstrap.cc:492] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: Bootstrap starting.
I20260619 16:55:48.091912 21173 tablet_bootstrap.cc:492] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: No bootstrap required, opened a new log
I20260619 16:55:48.091909 21172 tablet_bootstrap.cc:492] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: No bootstrap required, opened a new log
I20260619 16:55:48.092255 21173 ts_tablet_manager.cc:1397] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 16:55:48.092255 21172 ts_tablet_manager.cc:1397] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260619 16:55:48.092948 21174 tablet_bootstrap.cc:654] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:48.094398 21174 log.cc:824] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:48.097374 21174 tablet_bootstrap.cc:492] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: No bootstrap required, opened a new log
I20260619 16:55:48.097689 21174 ts_tablet_manager.cc:1397] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:55:48.103668 21172 raft_consensus.cc:348] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } }
I20260619 16:55:48.103972 21173 raft_consensus.cc:348] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } }
I20260619 16:55:48.104259 21172 raft_consensus.cc:374] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:48.104460 21173 raft_consensus.cc:374] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:48.104549 21173 raft_consensus.cc:729] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b1657361fe4eb18d7ad52a1da75b3c, State: Initialized, Role: FOLLOWER
I20260619 16:55:48.104549 21172 raft_consensus.cc:729] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6edd78e22746aebd3d9c799b5f1743, State: Initialized, Role: FOLLOWER
I20260619 16:55:48.105381 21173 consensus_queue.cc:260] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } }
I20260619 16:55:48.105379 21172 consensus_queue.cc:260] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } }
I20260619 16:55:48.105541 21174 raft_consensus.cc:348] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } }
I20260619 16:55:48.106048 21174 raft_consensus.cc:374] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:48.106168 21174 raft_consensus.cc:729] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afaa552bb0e4828a63b2ab4a82f77d1, State: Initialized, Role: FOLLOWER
I20260619 16:55:48.107276 21153 heartbeater.cc:503] Master 127.16.48.62:44691 was elected leader, sending a full tablet report...
I20260619 16:55:48.108263 21021 heartbeater.cc:503] Master 127.16.48.62:44691 was elected leader, sending a full tablet report...
I20260619 16:55:48.107029 21174 consensus_queue.cc:260] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } }
I20260619 16:55:48.108320 21173 ts_tablet_manager.cc:1428] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260619 16:55:48.108940 21172 ts_tablet_manager.cc:1428] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260619 16:55:48.110700 21174 ts_tablet_manager.cc:1428] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260619 16:55:48.151602 21179 raft_consensus.cc:482] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:48.152086 21179 raft_consensus.cc:504] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } }
I20260619 16:55:48.153950 21179 leader_election.cc:288] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df6edd78e22746aebd3d9c799b5f1743 (127.16.48.3:41083), 9afaa552bb0e4828a63b2ab4a82f77d1 (127.16.48.1:33283)
I20260619 16:55:48.161768 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.154498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:37823 (local address 127.16.48.1:33283)
0619 16:55:48.154771 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.154781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.155560 (+ 779us) server_negotiation.cc:407] Connection header received
0619 16:55:48.155886 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.155892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.155972 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.156062 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:48.156880 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.157678 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.159727 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.160583 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.160705 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.160787 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.161296 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.161445 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 16:55:48.161800 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.154896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:49741 (local address 127.16.48.3:41083)
0619 16:55:48.155082 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.155088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.155931 (+ 843us) server_negotiation.cc:407] Connection header received
0619 16:55:48.156086 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.156091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.156191 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.156295 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:55:48.157091 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.157911 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.159784 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.160814 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.160961 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.161062 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.161401 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.161542 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:55:48.161875 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.154218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:33283 (local address 127.16.48.2:37823)
0619 16:55:48.155429 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.155459 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:48.155709 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:48.156103 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:48.156108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:48.156168 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:48.156703 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.156717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.157812 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.157818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.159502 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.159517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.160861 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.160871 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.161088 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.161151 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 16:55:48.161233 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:55:48.161380 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":50,"spinlock_wait_cycles":28928,"thread_start_us":1267,"threads_started":1}
I20260619 16:55:48.161911 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.154389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:41083 (local address 127.16.48.2:49741)
0619 16:55:48.155799 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.155819 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:48.155945 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:48.156318 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:48.156325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:48.156354 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:55:48.156836 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.156849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.158057 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.158060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.159545 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:48.159558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.161035 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.161040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:48.161202 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.161260 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 16:55:48.161357 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:55:48.161482 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":3,"thread_start_us":706,"threads_started":1}
I20260619 16:55:48.163012 21108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01dfba8dfa43d69ff46c31ad2b4d55" candidate_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6edd78e22746aebd3d9c799b5f1743" is_pre_election: true
I20260619 16:55:48.163556 21108 raft_consensus.cc:2454] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b1657361fe4eb18d7ad52a1da75b3c in term 0.
I20260619 16:55:48.163450 20843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01dfba8dfa43d69ff46c31ad2b4d55" candidate_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" is_pre_election: true
I20260619 16:55:48.163965 20843 raft_consensus.cc:2454] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b1657361fe4eb18d7ad52a1da75b3c in term 0.
I20260619 16:55:48.164321 20911 leader_election.cc:302] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b1657361fe4eb18d7ad52a1da75b3c, df6edd78e22746aebd3d9c799b5f1743; no voters:
I20260619 16:55:48.164805 21179 raft_consensus.cc:2781] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:48.164945 21179 raft_consensus.cc:482] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:48.165058 21179 raft_consensus.cc:3037] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:48.167265 21179 raft_consensus.cc:504] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } }
I20260619 16:55:48.167951 21179 leader_election.cc:288] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [CANDIDATE]: Term 1 election: Requested vote from peers df6edd78e22746aebd3d9c799b5f1743 (127.16.48.3:41083), 9afaa552bb0e4828a63b2ab4a82f77d1 (127.16.48.1:33283)
I20260619 16:55:48.168507 21108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01dfba8dfa43d69ff46c31ad2b4d55" candidate_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6edd78e22746aebd3d9c799b5f1743"
I20260619 16:55:48.168522 20843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01dfba8dfa43d69ff46c31ad2b4d55" candidate_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1"
I20260619 16:55:48.168744 20843 raft_consensus.cc:3037] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:48.168758 21108 raft_consensus.cc:3037] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:48.170998 20843 raft_consensus.cc:2454] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b1657361fe4eb18d7ad52a1da75b3c in term 1.
I20260619 16:55:48.171006 21108 raft_consensus.cc:2454] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b1657361fe4eb18d7ad52a1da75b3c in term 1.
I20260619 16:55:48.171557 20911 leader_election.cc:302] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 59b1657361fe4eb18d7ad52a1da75b3c, 9afaa552bb0e4828a63b2ab4a82f77d1, df6edd78e22746aebd3d9c799b5f1743; no voters:
I20260619 16:55:48.171906 21179 raft_consensus.cc:2781] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:48.172587 21179 raft_consensus.cc:686] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 1 LEADER]: Becoming Leader. State: Replica: 59b1657361fe4eb18d7ad52a1da75b3c, State: Running, Role: LEADER
I20260619 16:55:48.173215 21179 consensus_queue.cc:237] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } }
W20260619 16:55:48.176817 20889 tablet.cc:2369] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:55:48.178541 20700 catalog_manager.cc:5495] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 59b1657361fe4eb18d7ad52a1da75b3c (127.16.48.2). New cstate: current_term: 1 leader_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59b1657361fe4eb18d7ad52a1da75b3c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 36365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 } health_report { overall_health: UNKNOWN } } }
W20260619 16:55:48.189536 21022 tablet.cc:2369] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:55:48.216063 21154 tablet.cc:2369] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:55:48.222059 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.216749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.16.48.2:36365)
0619 16:55:48.216943 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.216947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.216965 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:55:48.217190 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.217194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.217271 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.217400 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:55:48.217713 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.218397 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.219716 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.220272 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.220377 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.220853 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:55:48.221463 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:55:48.221637 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.221678 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.221748 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:55:48.230856 20843 raft_consensus.cc:1261] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [term 1 FOLLOWER]: Refusing update from remote peer 59b1657361fe4eb18d7ad52a1da75b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:48.230873 21108 raft_consensus.cc:1261] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [term 1 FOLLOWER]: Refusing update from remote peer 59b1657361fe4eb18d7ad52a1da75b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:48.231876 21183 consensus_queue.cc:1035] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9afaa552bb0e4828a63b2ab4a82f77d1" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:48.232316 21179 consensus_queue.cc:1035] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6edd78e22746aebd3d9c799b5f1743" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:48.243248 21186 mvcc.cc:205] Tried to move back new op lower bound from 7298613855143460864 to 7298613854922420224. Current Snapshot: MvccSnapshot[applied={T|T < 7298613855143460864}]
I20260619 16:55:48.244324 21188 mvcc.cc:205] Tried to move back new op lower bound from 7298613855143460864 to 7298613854922420224. Current Snapshot: MvccSnapshot[applied={T|T < 7298613855143460864}]
I20260619 16:55:48.263996 20699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:48.264389 20699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:48.268671 20699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18168eedfdbf4c0592d4d4115fb040b7: Sending DeleteTablet for 3 replicas of tablet 3e01dfba8dfa43d69ff46c31ad2b4d55
I20260619 16:55:48.269976 20823 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:48 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:59350
I20260619 16:55:48.270080 20956 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:48 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:60216
I20260619 16:55:48.270023 21088 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:48 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:43336
I20260619 16:55:48.272481 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 20761
I20260619 16:55:48.282649 21196 tablet_replica.cc:330] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: stopping tablet replica
I20260619 16:55:48.283300 21198 tablet_replica.cc:330] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: stopping tablet replica
I20260619 16:55:48.283972 21198 raft_consensus.cc:2229] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:48.284605 21196 raft_consensus.cc:2229] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:55:48.285260 21198 raft_consensus.cc:2258] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:48.285981 21196 raft_consensus.cc:2258] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:48.288743 21198 ts_tablet_manager.cc:1905] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:48.290930 21196 ts_tablet_manager.cc:1905] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:48.291572 21197 tablet_replica.cc:330] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: stopping tablet replica
I20260619 16:55:48.299723 21198 ts_tablet_manager.cc:1918] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:55:48.299762 21197 raft_consensus.cc:2229] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:48.300050 21198 log.cc:1196] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/3e01dfba8dfa43d69ff46c31ad2b4d55
I20260619 16:55:48.300375 21197 raft_consensus.cc:2258] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:48.300843 21198 ts_tablet_manager.cc:1939] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P df6edd78e22746aebd3d9c799b5f1743: Deleting consensus metadata
I20260619 16:55:48.302944 20686 catalog_manager.cc:4841] TS df6edd78e22746aebd3d9c799b5f1743 (127.16.48.3:41083): tablet 3e01dfba8dfa43d69ff46c31ad2b4d55 (table test-table [id=cba558b678de47d19e90d768281a73ad]) successfully deleted
Jun 19 16:55:48 dist-test-slave-3ffd krb5kdc[20656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:48.306989 21197 ts_tablet_manager.cc:1905] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:48.312273 21196 ts_tablet_manager.cc:1918] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:55:48.312503 21196 log.cc:1196] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/3e01dfba8dfa43d69ff46c31ad2b4d55
I20260619 16:55:48.313184 21196 ts_tablet_manager.cc:1939] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 59b1657361fe4eb18d7ad52a1da75b3c: Deleting consensus metadata
I20260619 16:55:48.315207 20686 catalog_manager.cc:4841] TS 59b1657361fe4eb18d7ad52a1da75b3c (127.16.48.2:36365): tablet 3e01dfba8dfa43d69ff46c31ad2b4d55 (table test-table [id=cba558b678de47d19e90d768281a73ad]) successfully deleted
I20260619 16:55:48.316756 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.274793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.16.48.1:33283)
0619 16:55:48.274975 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.274981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.292237 (+ 17256us) server_negotiation.cc:407] Connection header received
0619 16:55:48.292693 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.292698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.292809 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.292887 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:48.294523 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.295708 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.297671 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.298357 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.298456 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.307482 (+ 9026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:48.307499 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:48.307519 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:48.307580 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:48.312182 (+ 4602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:48.313358 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:48.313372 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:48.313377 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:48.313601 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:48.314123 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:48.314128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:48.314131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:48.314714 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:48.315188 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.315848 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.316214 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:55:48.316859 21197 ts_tablet_manager.cc:1918] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:55:48.317083 21197 log.cc:1196] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/3e01dfba8dfa43d69ff46c31ad2b4d55
I20260619 16:55:48.317516 20883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:48.430680 21197 ts_tablet_manager.cc:1939] T 3e01dfba8dfa43d69ff46c31ad2b4d55 P 9afaa552bb0e4828a63b2ab4a82f77d1: Deleting consensus metadata
I20260619 16:55:48.431412 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 20761
W20260619 16:55:48.453408 20684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.48.1:33283 (error 108)
I20260619 16:55:48.453537 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 20894
I20260619 16:55:48.455772 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.455230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:33283 (local address 127.0.0.1:59360)
0619 16:55:48.455461 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.455594 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:33283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260619 16:55:48.456081 20684 catalog_manager.cc:4584] TS 9afaa552bb0e4828a63b2ab4a82f77d1 (127.16.48.1:33283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e01dfba8dfa43d69ff46c31ad2b4d55: Network error: Client connection negotiation failed: client connection to 127.16.48.1:33283: connect: Connection refused (error 111)
Jun 19 16:55:48 dist-test-slave-3ffd krb5kdc[20656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:55:48.474279 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.454756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.16.48.2:36365)
0619 16:55:48.454909 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.454919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.454942 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:55:48.455005 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.455011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.455099 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.455227 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:48.456899 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.458166 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.460090 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.460909 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.461058 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.466603 (+ 5545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:48.466621 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:48.466647 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:48.466700 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:48.471245 (+ 4545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:48.472283 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:48.472291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:48.472293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:48.472443 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:48.472849 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:48.472855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:48.472858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:48.473209 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:48.473469 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.473771 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.473942 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:55:48.474967 21016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:48.509722 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.509181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:33283 (local address 127.0.0.1:59374)
0619 16:55:48.509409 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.509520 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:33283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 16:55:48.510828 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.510432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:33283 (local address 127.0.0.1:59382)
0619 16:55:48.510586 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.510673 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:33283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:55:48.571544 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.570976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:33283 (local address 127.0.0.1:59388)
0619 16:55:48.571239 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.571354 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:33283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 16:55:48.572786 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.572350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:33283 (local address 127.0.0.1:59402)
0619 16:55:48.572523 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.572616 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:33283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 16:55:48.591508 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 20894
I20260619 16:55:48.610122 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 21026
Jun 19 16:55:48 dist-test-slave-3ffd krb5kdc[20656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:55:48.626129 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.610895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.16.48.3:41083)
0619 16:55:48.611068 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:48.611072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:48.611091 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:55:48.611237 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:48.611241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:48.611306 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:48.611392 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:48.612796 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.613503 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.614823 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:48.615208 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:48.615309 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:48.619994 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:48.620015 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:48.620034 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:48.620085 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:48.623224 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:48.624027 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:48.624036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:48.624039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:48.624272 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:48.624631 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:48.624635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:48.624637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:48.625005 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:48.625343 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:48.625573 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 16:55:48.625758 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:55:48.626784 21148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:48.653573 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.653011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:33283 (local address 127.0.0.1:59410)
0619 16:55:48.653238 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.653360 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:33283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:55:48.654767 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:48.654311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:33283 (local address 127.0.0.1:59414)
0619 16:55:48.654519 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:48.654620 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:33283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 16:55:48.717903 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 21026
I20260619 16:55:48.734980 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 20669
I20260619 16:55:48.736177 20730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:48.843667 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 20669
2026-06-19T16:55:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4335 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T16:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:48Z Disabled control of system clock
I20260619 16:55:48.916759 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:35327
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:35327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:55:49.268755 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:49.269075 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:49.269244 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:49.276764 21208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:49.276892 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:49.276988 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:49.277081 21208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:49.277179 21208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:49.286886 21208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:35327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:35327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:49.288412 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:49.290361 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:49.303771 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:49.303686 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:49.303745 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:49.304239 21208 server_base.cc:1034] running on GCE node
I20260619 16:55:49.305378 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:49.308148 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:49.309427 21208 hybrid_clock.cc:648] HybridClock initialized: now 1781888149309365 us; error 61 us; skew 500 ppm
I20260619 16:55:49.309896 21208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:55:49.312598 21208 webserver.cc:460] Webserver started at http://127.16.48.62:45363/ using document root <none> and password file <none>
I20260619 16:55:49.313405 21208 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:49.313557 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:49.313910 21208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:49.317224 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "3c265dfc778449639eaa6188df0b2f04"
format_stamp: "Formatted at 2026-06-19 16:55:49 on dist-test-slave-3ffd"
server_key: "4f92534043a34540c625050db937b2b7"
server_key_iv: "998351ff416c387100fa620517bee76b"
server_key_version: "encryptionkey@0"
I20260619 16:55:49.318147 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "3c265dfc778449639eaa6188df0b2f04"
format_stamp: "Formatted at 2026-06-19 16:55:49 on dist-test-slave-3ffd"
server_key: "4f92534043a34540c625050db937b2b7"
server_key_iv: "998351ff416c387100fa620517bee76b"
server_key_version: "encryptionkey@0"
I20260619 16:55:49.324268 21208 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 16:55:49.328265 21222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:49.329958 21208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:49.330204 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "3c265dfc778449639eaa6188df0b2f04"
format_stamp: "Formatted at 2026-06-19 16:55:49 on dist-test-slave-3ffd"
server_key: "4f92534043a34540c625050db937b2b7"
server_key_iv: "998351ff416c387100fa620517bee76b"
server_key_version: "encryptionkey@0"
I20260619 16:55:49.330452 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:49.336493 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:49.337376 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:49.337698 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:49.356947 21208 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:35327
I20260619 16:55:49.356969 21273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:35327 every 8 connection(s)
I20260619 16:55:49.358505 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:49.359730 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 21208
I20260619 16:55:49.359937 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:49.360822 16576 external_mini_cluster.cc:1428] Setting key 65b8796a69896f6aec0f2f27931d989d
I20260619 16:55:49.364022 21274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:49.370419 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:49.363156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.16.48.62:35327)
0619 16:55:49.364340 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:49.364353 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:49.364404 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:55:49.366063 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:49.366088 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:49.366591 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:49.367088 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:49.368315 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:49.368321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:49.368335 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:49.368353 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:49.368457 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:49.368476 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:49.368762 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:49.368934 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:55:49.369242 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":402,"threads_started":1}
I20260619 16:55:49.376415 21274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04: Bootstrap starting.
I20260619 16:55:49.380210 21274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:49.381338 21274 log.cc:824] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:49.384454 21274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04: No bootstrap required, opened a new log
I20260619 16:55:49.392557 21274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c265dfc778449639eaa6188df0b2f04" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35327 } }
I20260619 16:55:49.392992 21274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:49.393095 21274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c265dfc778449639eaa6188df0b2f04, State: Initialized, Role: FOLLOWER
I20260619 16:55:49.393857 21274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c265dfc778449639eaa6188df0b2f04" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35327 } }
I20260619 16:55:49.394173 21274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:49.394336 21274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:49.394552 21274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:49.396903 21274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c265dfc778449639eaa6188df0b2f04" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35327 } }
I20260619 16:55:49.397432 21274 leader_election.cc:302] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c265dfc778449639eaa6188df0b2f04; no voters:
I20260619 16:55:49.397993 21274 leader_election.cc:288] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:49.398169 21279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:49.398873 21279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [term 1 LEADER]: Becoming Leader. State: Replica: 3c265dfc778449639eaa6188df0b2f04, State: Running, Role: LEADER
I20260619 16:55:49.399420 21279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c265dfc778449639eaa6188df0b2f04" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35327 } }
I20260619 16:55:49.400307 21274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:49.402427 21281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c265dfc778449639eaa6188df0b2f04. Latest consensus state: current_term: 1 leader_uuid: "3c265dfc778449639eaa6188df0b2f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c265dfc778449639eaa6188df0b2f04" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35327 } } }
I20260619 16:55:49.402894 21281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:49.404507 21280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c265dfc778449639eaa6188df0b2f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c265dfc778449639eaa6188df0b2f04" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35327 } } }
I20260619 16:55:49.404917 21280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:49.405706 21288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:49.411029 21288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:49.424078 21288 catalog_manager.cc:1347] Generated new cluster ID: 234c7e4fbd5a427da07a385f2ebd85ae
I20260619 16:55:49.424288 21288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:49.435117 21288 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:49.435612 21288 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:49.456053 21288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04: Generated new TSK 0
I20260619 16:55:49.457306 21288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:55:49.473937 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:35327
--builtin_ntp_servers=127.16.48.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:55:49.821735 21298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:49.822047 21298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:49.822204 21298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:49.829780 21298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:49.829911 21298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:49.830099 21298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:49.839752 21298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:35327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:49.841321 21298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:49.843227 21298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:49.857223 21303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:49.857190 21304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:49.857349 21306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:49.858798 21298 server_base.cc:1034] running on GCE node
I20260619 16:55:49.859544 21298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:49.860713 21298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:49.861979 21298 hybrid_clock.cc:648] HybridClock initialized: now 1781888149861895 us; error 82 us; skew 500 ppm
I20260619 16:55:49.862353 21298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:55:49.865100 21298 webserver.cc:460] Webserver started at http://127.16.48.1:37883/ using document root <none> and password file <none>
I20260619 16:55:49.865921 21298 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:49.866060 21298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:49.866415 21298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:49.869705 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "003a3aecf7844a299e776067fc26b51f"
format_stamp: "Formatted at 2026-06-19 16:55:49 on dist-test-slave-3ffd"
server_key: "3e6784fcd44dba0dde9ffdcbe3238137"
server_key_iv: "e9637339033cdae42d51c66fb231d5fd"
server_key_version: "encryptionkey@0"
I20260619 16:55:49.870621 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "003a3aecf7844a299e776067fc26b51f"
format_stamp: "Formatted at 2026-06-19 16:55:49 on dist-test-slave-3ffd"
server_key: "3e6784fcd44dba0dde9ffdcbe3238137"
server_key_iv: "e9637339033cdae42d51c66fb231d5fd"
server_key_version: "encryptionkey@0"
I20260619 16:55:49.876478 21298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:55:49.880455 21312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:49.881990 21298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:49.882200 21298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "003a3aecf7844a299e776067fc26b51f"
format_stamp: "Formatted at 2026-06-19 16:55:49 on dist-test-slave-3ffd"
server_key: "3e6784fcd44dba0dde9ffdcbe3238137"
server_key_iv: "e9637339033cdae42d51c66fb231d5fd"
server_key_version: "encryptionkey@0"
I20260619 16:55:49.882438 21298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:49.888432 21298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:49.889308 21298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:49.889583 21298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:49.890547 21298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:49.891990 21298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:49.892185 21298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:49.892318 21298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:49.892400 21298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:49.928437 21298 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:37973
I20260619 16:55:49.928493 21424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:37973 every 8 connection(s)
I20260619 16:55:49.929874 21298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:49.935468 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 21298
I20260619 16:55:49.935679 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:49.936362 16576 external_mini_cluster.cc:1428] Setting key 144daed6fe679027f4b5d7e1c909ab1d
I20260619 16:55:49.938977 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:35327
--builtin_ntp_servers=127.16.48.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:55:49.939081 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:49.932777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:46125 (local address 127.16.48.62:35327)
0619 16:55:49.933248 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:49.933252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:49.934627 (+ 1375us) server_negotiation.cc:407] Connection header received
0619 16:55:49.936673 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:49.936678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:49.936752 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:49.936875 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:49.938291 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:49.938297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:49.938301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:49.938321 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:49.938368 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:49.938380 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:49.938507 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:49.938720 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 16:55:49.938787 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260619 16:55:49.940191 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:49.933389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:35327 (local address 127.16.48.1:46125)
0619 16:55:49.934413 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:49.934470 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:49.936340 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:49.937157 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:49.937168 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:49.937276 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:49.937984 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:55:49.937986 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:49.938028 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:55:49.938119 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:49.938470 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:49.938483 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:49.938493 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:55:49.938809 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 16:55:49.939110 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":284,"threads_started":1}
I20260619 16:55:49.942040 21425 heartbeater.cc:347] Connected to a master server at 127.16.48.62:35327
I20260619 16:55:49.942386 21425 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:49.943024 21425 heartbeater.cc:511] Master 127.16.48.62:35327 requested a full tablet report, sending...
I20260619 16:55:49.945405 21239 ts_manager.cc:194] Registered new tserver with Master: 003a3aecf7844a299e776067fc26b51f (127.16.48.1:37973)
W20260619 16:55:50.289120 21429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:50.289453 21429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:50.289608 21429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:50.297405 21429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:50.297530 21429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:50.297755 21429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:50.307785 21429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:35327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:50.309193 21429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:50.311175 21429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:50.325783 21437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:50.325881 21435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:50.325826 21434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:50.330008 21429 server_base.cc:1034] running on GCE node
I20260619 16:55:50.330781 21429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:50.331848 21429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:50.333091 21429 hybrid_clock.cc:648] HybridClock initialized: now 1781888150333005 us; error 88 us; skew 500 ppm
I20260619 16:55:50.333452 21429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:55:50.335954 21429 webserver.cc:460] Webserver started at http://127.16.48.2:34427/ using document root <none> and password file <none>
I20260619 16:55:50.336809 21429 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:50.336956 21429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:50.337321 21429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:50.340656 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "3748f235950d4738b661fb60492ed4bd"
format_stamp: "Formatted at 2026-06-19 16:55:50 on dist-test-slave-3ffd"
server_key: "4570402f53d125bad01f0587faadf033"
server_key_iv: "3eadeea42c83533d323d5bc38ce45b0a"
server_key_version: "encryptionkey@0"
I20260619 16:55:50.341580 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "3748f235950d4738b661fb60492ed4bd"
format_stamp: "Formatted at 2026-06-19 16:55:50 on dist-test-slave-3ffd"
server_key: "4570402f53d125bad01f0587faadf033"
server_key_iv: "3eadeea42c83533d323d5bc38ce45b0a"
server_key_version: "encryptionkey@0"
I20260619 16:55:50.347568 21429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 16:55:50.351640 21443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:50.353325 21429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:55:50.353551 21429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "3748f235950d4738b661fb60492ed4bd"
format_stamp: "Formatted at 2026-06-19 16:55:50 on dist-test-slave-3ffd"
server_key: "4570402f53d125bad01f0587faadf033"
server_key_iv: "3eadeea42c83533d323d5bc38ce45b0a"
server_key_version: "encryptionkey@0"
I20260619 16:55:50.353817 21429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:50.360592 21429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:50.361595 21429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:50.361901 21429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:50.363058 21429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:50.364792 21429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:50.364915 21429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:50.365054 21429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:50.365139 21429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:50.399732 21429 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:35989
I20260619 16:55:50.399791 21555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:35989 every 8 connection(s)
I20260619 16:55:50.401346 21429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:50.402698 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 21429
I20260619 16:55:50.402870 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:50.403436 16576 external_mini_cluster.cc:1428] Setting key 6f5a6a0579fb0f90fa352fadd087da19
I20260619 16:55:50.406780 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:35327
--builtin_ntp_servers=127.16.48.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:55:50.412256 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.404652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:56351 (local address 127.16.48.62:35327)
0619 16:55:50.405722 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:50.405728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:50.406674 (+ 946us) server_negotiation.cc:407] Connection header received
0619 16:55:50.409641 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:50.409646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:50.409701 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:50.409784 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:50.411475 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:50.411479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:50.411482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:50.411499 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:50.411528 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:50.411540 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:50.411639 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:50.411903 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 16:55:50.411957 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939}
I20260619 16:55:50.413657 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.405248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:35327 (local address 127.16.48.2:56351)
0619 16:55:50.406493 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:50.406556 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:50.409252 (+ 2696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:50.410114 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:50.410131 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:50.410279 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:50.411073 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:55:50.411079 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:50.411138 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:55:50.411269 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:50.411637 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:50.411650 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:50.411661 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 16:55:50.411968 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 16:55:50.412331 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":371,"threads_started":1}
I20260619 16:55:50.415201 21556 heartbeater.cc:347] Connected to a master server at 127.16.48.62:35327
I20260619 16:55:50.415539 21556 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:50.416344 21556 heartbeater.cc:511] Master 127.16.48.62:35327 requested a full tablet report, sending...
I20260619 16:55:50.418131 21239 ts_manager.cc:194] Registered new tserver with Master: 3748f235950d4738b661fb60492ed4bd (127.16.48.2:35989)
W20260619 16:55:50.757962 21559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:50.758301 21559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:50.758468 21559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:50.766203 21559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:50.766333 21559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:50.766511 21559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:50.775992 21559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:35327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:50.777482 21559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:50.779419 21559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:50.793955 21564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:50.793941 21565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:50.793988 21567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:50.794534 21559 server_base.cc:1034] running on GCE node
I20260619 16:55:50.795404 21559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:50.796483 21559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:50.797744 21559 hybrid_clock.cc:648] HybridClock initialized: now 1781888150797675 us; error 62 us; skew 500 ppm
I20260619 16:55:50.798103 21559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:55:50.800806 21559 webserver.cc:460] Webserver started at http://127.16.48.3:43515/ using document root <none> and password file <none>
I20260619 16:55:50.801676 21559 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:50.801811 21559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:50.802150 21559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:50.805545 21559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "f6ac7674e1294b8788b7d6886d719a85"
format_stamp: "Formatted at 2026-06-19 16:55:50 on dist-test-slave-3ffd"
server_key: "aec6f3cf2d73bf8ac6c6b51a8deae1ce"
server_key_iv: "5766e7e8a40f62dfb67c1c4a53f1ec75"
server_key_version: "encryptionkey@0"
I20260619 16:55:50.806435 21559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "f6ac7674e1294b8788b7d6886d719a85"
format_stamp: "Formatted at 2026-06-19 16:55:50 on dist-test-slave-3ffd"
server_key: "aec6f3cf2d73bf8ac6c6b51a8deae1ce"
server_key_iv: "5766e7e8a40f62dfb67c1c4a53f1ec75"
server_key_version: "encryptionkey@0"
I20260619 16:55:50.812105 21559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:55:50.816071 21573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:50.817782 21559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:50.818110 21559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "f6ac7674e1294b8788b7d6886d719a85"
format_stamp: "Formatted at 2026-06-19 16:55:50 on dist-test-slave-3ffd"
server_key: "aec6f3cf2d73bf8ac6c6b51a8deae1ce"
server_key_iv: "5766e7e8a40f62dfb67c1c4a53f1ec75"
server_key_version: "encryptionkey@0"
I20260619 16:55:50.818382 21559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:50.824851 21559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:50.825793 21559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:50.826092 21559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:50.827083 21559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:50.828711 21559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:50.828837 21559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:50.828979 21559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:50.829066 21559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:50.863025 21559 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:39035
I20260619 16:55:50.863075 21685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:39035 every 8 connection(s)
I20260619 16:55:50.864485 21559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:50.872027 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 21559
I20260619 16:55:50.872246 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:50.872701 16576 external_mini_cluster.cc:1428] Setting key 84ecd9e5075995a0ecec9f30a7c0cbe4
I20260619 16:55:50.873627 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.867379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:59555 (local address 127.16.48.62:35327)
0619 16:55:50.867568 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:50.867572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:50.869256 (+ 1684us) server_negotiation.cc:407] Connection header received
0619 16:55:50.871015 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:50.871021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:50.871104 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:50.871221 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:50.872828 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:50.872833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:50.872837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:50.872856 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:50.872888 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:50.872901 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:50.873032 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:50.873235 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 16:55:50.873307 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:55:50.874809 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.867996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:35327 (local address 127.16.48.3:59555)
0619 16:55:50.869068 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:50.869108 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:50.870710 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:50.871472 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:50.871481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:50.871591 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:50.872491 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:55:50.872494 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:50.872537 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:55:50.872633 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:50.873006 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:50.873017 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:50.873025 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:55:50.873318 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 16:55:50.873653 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":306,"threads_started":1}
I20260619 16:55:50.876214 21686 heartbeater.cc:347] Connected to a master server at 127.16.48.62:35327
I20260619 16:55:50.876544 21686 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:50.877118 21686 heartbeater.cc:511] Master 127.16.48.62:35327 requested a full tablet report, sending...
I20260619 16:55:50.878549 21239 ts_manager.cc:194] Registered new tserver with Master: f6ac7674e1294b8788b7d6886d719a85 (127.16.48.3:39035)
I20260619 16:55:50.888556 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:55:50.895936 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.893136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.16.48.62:35327)
0619 16:55:50.893378 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:50.893387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:50.893905 (+ 518us) server_negotiation.cc:407] Connection header received
0619 16:55:50.894077 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:50.894081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:50.894144 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:50.894254 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:50.895155 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:50.895159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:50.895162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:50.895180 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:50.895215 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:50.895229 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:50.895346 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:50.895440 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 16:55:50.895498 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:55:50.900821 21239 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:55:50.908768 21239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:55:50.932476 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.924507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.16.48.1:37973)
0619 16:55:50.925099 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:50.925112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:50.928820 (+ 3708us) server_negotiation.cc:407] Connection header received
0619 16:55:50.929044 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:50.929053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:50.929279 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:50.929448 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:50.931432 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:50.931441 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:50.931458 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:50.931484 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:50.931555 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:50.931580 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:50.931801 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:50.931889 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 16:55:50.932001 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
I20260619 16:55:50.932593 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.923707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:37973 (local address 127.0.0.1:49532)
0619 16:55:50.928659 (+ 4952us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:50.928703 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:50.928868 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:50.929454 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:50.929462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:50.929566 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:50.930796 (+ 1230us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:55:50.930799 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:50.930831 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:55:50.930925 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:50.931702 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:50.931715 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:50.931725 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:55:50.931905 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 16:55:50.932004 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4720,"threads_started":1}
I20260619 16:55:50.935976 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.923979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:35989 (local address 127.0.0.1:51336)
0619 16:55:50.932795 (+ 8816us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:50.932814 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:50.932905 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:50.933526 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:50.933530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:50.933556 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:50.934192 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:55:50.934196 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:50.934214 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:55:50.934262 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:50.935537 (+ 1275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:50.935541 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:50.935546 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:55:50.935647 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:55:50.935706 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8725,"threads_started":1}
I20260619 16:55:50.936637 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.924579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.16.48.2:35989)
0619 16:55:50.925341 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:50.925348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:50.933042 (+ 7694us) server_negotiation.cc:407] Connection header received
0619 16:55:50.933126 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:50.933134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:50.933364 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:50.933534 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:50.935171 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:50.935179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:50.935193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:50.935212 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:50.935281 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:50.935310 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:50.935444 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:50.936242 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 16:55:50.936359 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":453,"threads_started":1}
I20260619 16:55:50.940920 21360 tablet_service.cc:1467] Processing CreateTablet for tablet 8d77106bbe7741c68c97b31895fa9971 (DEFAULT_TABLE table=test-table [id=2d14b2e73d704b08baa33ce4c9fe3423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:50.942701 21360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d77106bbe7741c68c97b31895fa9971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:50.945413 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.941426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:39035 (local address 127.0.0.1:37692)
0619 16:55:50.941657 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:50.941678 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:50.941785 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:50.942952 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:50.942955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:50.942977 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:50.943662 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:55:50.943664 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:50.943679 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:55:50.943720 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:50.944809 (+ 1089us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:50.944815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:50.944820 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:55:50.944953 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 16:55:50.945015 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:55:50.945448 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:50.941683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.16.48.3:39035)
0619 16:55:50.942339 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:50.942348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:50.942381 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:55:50.942462 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:50.942471 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:50.942718 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:50.943369 (+ 651us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:50.944609 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:50.944614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:50.944625 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:50.944639 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:50.944682 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:50.944695 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:50.944891 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:50.944962 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 16:55:50.945057 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":312,"threads_started":1}
I20260619 16:55:50.945838 21491 tablet_service.cc:1467] Processing CreateTablet for tablet 8d77106bbe7741c68c97b31895fa9971 (DEFAULT_TABLE table=test-table [id=2d14b2e73d704b08baa33ce4c9fe3423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:50.947417 21491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d77106bbe7741c68c97b31895fa9971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:50.948671 21425 heartbeater.cc:503] Master 127.16.48.62:35327 was elected leader, sending a full tablet report...
I20260619 16:55:50.951717 21621 tablet_service.cc:1467] Processing CreateTablet for tablet 8d77106bbe7741c68c97b31895fa9971 (DEFAULT_TABLE table=test-table [id=2d14b2e73d704b08baa33ce4c9fe3423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:55:50.953025 21621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d77106bbe7741c68c97b31895fa9971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:50.958606 21703 tablet_bootstrap.cc:492] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: Bootstrap starting.
I20260619 16:55:50.962759 21703 tablet_bootstrap.cc:654] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:50.964157 21704 tablet_bootstrap.cc:492] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: Bootstrap starting.
I20260619 16:55:50.965183 21703 log.cc:824] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:50.967602 21706 tablet_bootstrap.cc:492] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: Bootstrap starting.
I20260619 16:55:50.968186 21704 tablet_bootstrap.cc:654] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:50.969027 21703 tablet_bootstrap.cc:492] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: No bootstrap required, opened a new log
I20260619 16:55:50.969328 21703 ts_tablet_manager.cc:1397] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 16:55:50.969601 21704 log.cc:824] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:50.971422 21706 tablet_bootstrap.cc:654] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:50.972795 21706 log.cc:824] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:50.973161 21704 tablet_bootstrap.cc:492] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: No bootstrap required, opened a new log
I20260619 16:55:50.973515 21704 ts_tablet_manager.cc:1397] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260619 16:55:50.976588 21706 tablet_bootstrap.cc:492] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: No bootstrap required, opened a new log
I20260619 16:55:50.976931 21706 ts_tablet_manager.cc:1397] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 16:55:50.978317 21703 raft_consensus.cc:348] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } }
I20260619 16:55:50.978755 21703 raft_consensus.cc:374] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:50.978895 21703 raft_consensus.cc:729] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003a3aecf7844a299e776067fc26b51f, State: Initialized, Role: FOLLOWER
I20260619 16:55:50.979557 21703 consensus_queue.cc:260] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } }
I20260619 16:55:50.982694 21703 ts_tablet_manager.cc:1428] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260619 16:55:50.984611 21704 raft_consensus.cc:348] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } }
I20260619 16:55:50.985105 21704 raft_consensus.cc:374] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:50.985241 21704 raft_consensus.cc:729] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748f235950d4738b661fb60492ed4bd, State: Initialized, Role: FOLLOWER
I20260619 16:55:50.985929 21704 consensus_queue.cc:260] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } }
I20260619 16:55:50.987367 21556 heartbeater.cc:503] Master 127.16.48.62:35327 was elected leader, sending a full tablet report...
I20260619 16:55:50.987993 21706 raft_consensus.cc:348] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } }
I20260619 16:55:50.988480 21706 raft_consensus.cc:374] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:50.988583 21706 raft_consensus.cc:729] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ac7674e1294b8788b7d6886d719a85, State: Initialized, Role: FOLLOWER
I20260619 16:55:50.989020 21704 ts_tablet_manager.cc:1428] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260619 16:55:50.989382 21706 consensus_queue.cc:260] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } }
I20260619 16:55:50.990607 21686 heartbeater.cc:503] Master 127.16.48.62:35327 was elected leader, sending a full tablet report...
I20260619 16:55:50.991313 21706 ts_tablet_manager.cc:1428] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 16:55:51.032829 21711 raft_consensus.cc:482] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:55:51.033142 21711 raft_consensus.cc:504] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } }
I20260619 16:55:51.034912 21711 leader_election.cc:288] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 003a3aecf7844a299e776067fc26b51f (127.16.48.1:37973), 3748f235950d4738b661fb60492ed4bd (127.16.48.2:35989)
I20260619 16:55:51.037771 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:51.035113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:37973 (local address 127.16.48.3:59797)
0619 16:55:51.035339 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:51.035364 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:51.035483 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:51.035967 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:51.035974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:51.036009 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:51.036828 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:55:51.036833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:51.036858 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:55:51.036915 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:51.037319 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:51.037324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:51.037329 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:55:51.037473 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:55:51.037524 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 16:55:51.037791 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:51.035375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:59797 (local address 127.16.48.1:37973)
0619 16:55:51.035584 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:51.035590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:51.035616 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:55:51.035689 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:51.035697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:51.035762 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:51.035908 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:51.037094 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:51.037098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:51.037101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:51.037143 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:51.037185 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:51.037203 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:51.037335 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:51.037468 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 16:55:51.037516 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 16:55:51.038347 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:51.035616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:37477 (local address 127.16.48.2:35989)
0619 16:55:51.035772 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:51.035781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:51.036407 (+ 626us) server_negotiation.cc:407] Connection header received
0619 16:55:51.036503 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:51.036509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:51.036574 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:51.036673 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:51.037665 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:51.037671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:51.037676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:51.037695 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:51.037727 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:51.037741 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:51.037867 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:51.038027 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 16:55:51.038093 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:55:51.039072 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:51.035441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:35989 (local address 127.16.48.3:37477)
0619 16:55:51.036284 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:51.036303 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:51.036407 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:51.036743 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:51.036749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:51.036781 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:51.037462 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:55:51.037465 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:51.037480 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:55:51.037517 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:51.037853 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:51.037858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:51.037864 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:55:51.038606 (+ 742us) client_negotiation.cc:240] Negotiation successful
0619 16:55:51.038660 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":490,"threads_started":1}
I20260619 16:55:51.038942 21380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d77106bbe7741c68c97b31895fa9971" candidate_uuid: "f6ac7674e1294b8788b7d6886d719a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003a3aecf7844a299e776067fc26b51f" is_pre_election: true
I20260619 16:55:51.039597 21380 raft_consensus.cc:2454] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ac7674e1294b8788b7d6886d719a85 in term 0.
I20260619 16:55:51.040203 21511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d77106bbe7741c68c97b31895fa9971" candidate_uuid: "f6ac7674e1294b8788b7d6886d719a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748f235950d4738b661fb60492ed4bd" is_pre_election: true
I20260619 16:55:51.040320 21574 leader_election.cc:302] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003a3aecf7844a299e776067fc26b51f, f6ac7674e1294b8788b7d6886d719a85; no voters:
I20260619 16:55:51.040699 21711 raft_consensus.cc:2781] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:55:51.040742 21511 raft_consensus.cc:2454] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ac7674e1294b8788b7d6886d719a85 in term 0.
I20260619 16:55:51.040863 21711 raft_consensus.cc:482] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:55:51.040959 21711 raft_consensus.cc:3037] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:51.043303 21711 raft_consensus.cc:504] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } }
I20260619 16:55:51.043893 21711 leader_election.cc:288] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [CANDIDATE]: Term 1 election: Requested vote from peers 003a3aecf7844a299e776067fc26b51f (127.16.48.1:37973), 3748f235950d4738b661fb60492ed4bd (127.16.48.2:35989)
I20260619 16:55:51.044379 21380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d77106bbe7741c68c97b31895fa9971" candidate_uuid: "f6ac7674e1294b8788b7d6886d719a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003a3aecf7844a299e776067fc26b51f"
I20260619 16:55:51.044442 21511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d77106bbe7741c68c97b31895fa9971" candidate_uuid: "f6ac7674e1294b8788b7d6886d719a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748f235950d4738b661fb60492ed4bd"
I20260619 16:55:51.044622 21380 raft_consensus.cc:3037] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:51.044677 21511 raft_consensus.cc:3037] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:51.046918 21511 raft_consensus.cc:2454] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ac7674e1294b8788b7d6886d719a85 in term 1.
I20260619 16:55:51.046916 21380 raft_consensus.cc:2454] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ac7674e1294b8788b7d6886d719a85 in term 1.
I20260619 16:55:51.047420 21574 leader_election.cc:302] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003a3aecf7844a299e776067fc26b51f, f6ac7674e1294b8788b7d6886d719a85; no voters:
I20260619 16:55:51.047761 21711 raft_consensus.cc:2781] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:51.048285 21711 raft_consensus.cc:686] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 1 LEADER]: Becoming Leader. State: Replica: f6ac7674e1294b8788b7d6886d719a85, State: Running, Role: LEADER
I20260619 16:55:51.048857 21711 consensus_queue.cc:237] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } }
I20260619 16:55:51.053831 21238 catalog_manager.cc:5495] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6ac7674e1294b8788b7d6886d719a85 (127.16.48.3). New cstate: current_term: 1 leader_uuid: "f6ac7674e1294b8788b7d6886d719a85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ac7674e1294b8788b7d6886d719a85" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 39035 } health_report { overall_health: HEALTHY } } }
I20260619 16:55:51.089174 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:51.086776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.16.48.3:39035)
0619 16:55:51.086992 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:51.086999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:51.087115 (+ 116us) server_negotiation.cc:407] Connection header received
0619 16:55:51.087395 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:51.087399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:51.087492 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:51.087609 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:51.088551 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:51.088559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:51.088565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:51.088589 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:51.088622 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:51.088635 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:51.088771 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:51.088855 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 16:55:51.088918 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:55:51.098156 21511 raft_consensus.cc:1261] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [term 1 FOLLOWER]: Refusing update from remote peer f6ac7674e1294b8788b7d6886d719a85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:51.098158 21380 raft_consensus.cc:1261] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [term 1 FOLLOWER]: Refusing update from remote peer f6ac7674e1294b8788b7d6886d719a85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:55:51.099205 21713 consensus_queue.cc:1035] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748f235950d4738b661fb60492ed4bd" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:51.099571 21711 consensus_queue.cc:1035] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "003a3aecf7844a299e776067fc26b51f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:55:51.114418 21718 mvcc.cc:205] Tried to move back new op lower bound from 7298613866887471104 to 7298613866700263424. Current Snapshot: MvccSnapshot[applied={T|T < 7298613866887471104}]
I20260619 16:55:51.118649 21717 mvcc.cc:205] Tried to move back new op lower bound from 7298613866887471104 to 7298613866700263424. Current Snapshot: MvccSnapshot[applied={T|T < 7298613866887471104}]
I20260619 16:55:51.128676 21238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:51.129007 21238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:55:51.132999 21238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c265dfc778449639eaa6188df0b2f04: Sending DeleteTablet for 3 replicas of tablet 8d77106bbe7741c68c97b31895fa9971
I20260619 16:55:51.134387 21360 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d77106bbe7741c68c97b31895fa9971 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:51 UTC) from {username='slave'} at 127.0.0.1:49532
I20260619 16:55:51.134819 21491 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d77106bbe7741c68c97b31895fa9971 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:51 UTC) from {username='slave'} at 127.0.0.1:51336
I20260619 16:55:51.135470 21621 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d77106bbe7741c68c97b31895fa9971 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:55:51 UTC) from {username='slave'} at 127.0.0.1:37692
I20260619 16:55:51.136487 21726 tablet_replica.cc:330] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: stopping tablet replica
I20260619 16:55:51.136749 21727 tablet_replica.cc:330] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: stopping tablet replica
I20260619 16:55:51.137072 21726 raft_consensus.cc:2229] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:51.137109 21727 raft_consensus.cc:2229] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:55:51.137188 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 21298
I20260619 16:55:51.137663 21726 raft_consensus.cc:2258] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:51.137856 21727 raft_consensus.cc:2258] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:51.138614 21725 tablet_replica.cc:330] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: stopping tablet replica
I20260619 16:55:51.139070 21725 raft_consensus.cc:2229] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:55:51.139606 21725 raft_consensus.cc:2258] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:55:51.141391 21727 ts_tablet_manager.cc:1905] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:51.141680 21726 ts_tablet_manager.cc:1905] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:51.141692 21725 ts_tablet_manager.cc:1905] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:55:51.149178 21727 ts_tablet_manager.cc:1918] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:55:51.149163 21726 ts_tablet_manager.cc:1918] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:55:51.149164 21725 ts_tablet_manager.cc:1918] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:55:51.149374 21725 log.cc:1196] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/8d77106bbe7741c68c97b31895fa9971
I20260619 16:55:51.149382 21727 log.cc:1196] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/8d77106bbe7741c68c97b31895fa9971
I20260619 16:55:51.149600 21726 log.cc:1196] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/8d77106bbe7741c68c97b31895fa9971
I20260619 16:55:51.149998 21727 ts_tablet_manager.cc:1939] T 8d77106bbe7741c68c97b31895fa9971 P f6ac7674e1294b8788b7d6886d719a85: Deleting consensus metadata
I20260619 16:55:51.149994 21725 ts_tablet_manager.cc:1939] T 8d77106bbe7741c68c97b31895fa9971 P 003a3aecf7844a299e776067fc26b51f: Deleting consensus metadata
I20260619 16:55:51.150307 21726 ts_tablet_manager.cc:1939] T 8d77106bbe7741c68c97b31895fa9971 P 3748f235950d4738b661fb60492ed4bd: Deleting consensus metadata
I20260619 16:55:51.151935 21223 catalog_manager.cc:4841] TS 003a3aecf7844a299e776067fc26b51f (127.16.48.1:37973): tablet 8d77106bbe7741c68c97b31895fa9971 (table test-table [id=2d14b2e73d704b08baa33ce4c9fe3423]) successfully deleted
I20260619 16:55:51.152192 21224 catalog_manager.cc:4841] TS f6ac7674e1294b8788b7d6886d719a85 (127.16.48.3:39035): tablet 8d77106bbe7741c68c97b31895fa9971 (table test-table [id=2d14b2e73d704b08baa33ce4c9fe3423]) successfully deleted
I20260619 16:55:51.152614 21224 catalog_manager.cc:4841] TS 3748f235950d4738b661fb60492ed4bd (127.16.48.2:35989): tablet 8d77106bbe7741c68c97b31895fa9971 (table test-table [id=2d14b2e73d704b08baa33ce4c9fe3423]) successfully deleted
I20260619 16:55:51.154155 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:51.138793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.16.48.1:37973)
0619 16:55:51.139079 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:51.139087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:51.139555 (+ 468us) server_negotiation.cc:407] Connection header received
0619 16:55:51.151807 (+ 12252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:51.151813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:51.151947 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:51.152037 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:51.153461 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:51.153515 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:51.153522 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:51.153548 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:51.153590 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:51.153608 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:51.153754 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:51.153805 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:55:51.153866 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260619 16:55:51.154927 21420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:51.245579 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 21298
I20260619 16:55:51.263444 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 21429
I20260619 16:55:51.272048 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:51.264439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.16.48.2:35989)
0619 16:55:51.264641 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:51.264648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:51.264672 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:55:51.264741 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:51.264746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:51.264853 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:51.264966 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:51.271315 (+ 6349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:51.271323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:51.271328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:51.271357 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:51.271408 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:51.271428 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:51.271604 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:51.271655 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:55:51.271722 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:55:51.272913 21551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:51.368569 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 21429
I20260619 16:55:51.385875 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 21559
I20260619 16:55:51.389295 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:51.386760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.16.48.3:39035)
0619 16:55:51.386945 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:51.386951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:51.386971 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:55:51.387190 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:51.387195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:51.387246 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:51.387368 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:51.388455 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:51.388459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:51.388463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:51.388479 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:51.388536 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:51.388557 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:51.388746 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:51.388872 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 16:55:51.388940 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:55:51.390199 21681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:51.508751 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 21559
I20260619 16:55:51.529567 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 21208
I20260619 16:55:51.530697 21269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:51.638690 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 21208
2026-06-19T16:55:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2793 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 16:55:51.682214 16576 mini_oidc.cc:152] Starting JWKS server
I20260619 16:55:51.685235 16576 webserver.cc:460] Webserver started at https://127.0.0.1:37331/ using document root <none> and password file <none>
I20260619 16:55:51.685621 16576 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:55:51.686318 16576 webserver.cc:460] Webserver started at http://127.0.0.1:33623/ using document root <none> and password file <none>
2026-06-19T16:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:51Z Disabled control of system clock
I20260619 16:55:51.715906 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:38427
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:43259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:38427
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37331/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/testchainca.cert with env {}
W20260619 16:55:52.056468 21738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:52.056865 21738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:52.057041 21738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:52.064612 21738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:52.064800 21738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:52.064889 21738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:52.064992 21738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:52.065073 21738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:55:52.065165 21738 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:55:52.065246 21738 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37331/jwks.json
I20260619 16:55:52.075582 21738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:43259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:38427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:38427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37331/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:52.077064 21738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:52.078446 21738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:52.091478 21744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:52.091480 21746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:52.091537 21743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:52.091849 21738 server_base.cc:1034] running on GCE node
I20260619 16:55:52.093040 21738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:52.094607 21738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:52.095814 21738 hybrid_clock.cc:648] HybridClock initialized: now 1781888152095773 us; error 37 us; skew 500 ppm
I20260619 16:55:52.096238 21738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:55:52.098621 21738 webserver.cc:460] Webserver started at http://127.16.48.62:36263/ using document root <none> and password file <none>
I20260619 16:55:52.099349 21738 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:52.099475 21738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:52.099812 21738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:52.102905 21738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "3820cb27032142f2a49e08d7cb71aa49"
format_stamp: "Formatted at 2026-06-19 16:55:52 on dist-test-slave-3ffd"
server_key: "d7f0eb28be1140b995d8a8da5ec12d53"
server_key_iv: "ccc2a0ddfa89bc11d10a7166ffde49dc"
server_key_version: "encryptionkey@0"
I20260619 16:55:52.103752 21738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "3820cb27032142f2a49e08d7cb71aa49"
format_stamp: "Formatted at 2026-06-19 16:55:52 on dist-test-slave-3ffd"
server_key: "d7f0eb28be1140b995d8a8da5ec12d53"
server_key_iv: "ccc2a0ddfa89bc11d10a7166ffde49dc"
server_key_version: "encryptionkey@0"
I20260619 16:55:52.109130 21738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:55:52.112895 21752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:52.114348 21738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:52.114550 21738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "3820cb27032142f2a49e08d7cb71aa49"
format_stamp: "Formatted at 2026-06-19 16:55:52 on dist-test-slave-3ffd"
server_key: "d7f0eb28be1140b995d8a8da5ec12d53"
server_key_iv: "ccc2a0ddfa89bc11d10a7166ffde49dc"
server_key_version: "encryptionkey@0"
I20260619 16:55:52.114806 21738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:52.133945 21738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:52.134895 21738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:52.135239 21738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:52.154625 21738 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:38427
I20260619 16:55:52.154631 21803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:38427 every 8 connection(s)
I20260619 16:55:52.156258 21738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:52.157034 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 21738
I20260619 16:55:52.157279 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:52.158129 16576 external_mini_cluster.cc:1428] Setting key fddac102943b6a93bff282f074eb0779
I20260619 16:55:52.161588 21804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:52.174350 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:52.160475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.16.48.62:38427)
0619 16:55:52.161615 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:52.161628 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:52.161713 (+ 85us) server_negotiation.cc:407] Connection header received
0619 16:55:52.163420 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:52.163444 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:52.163954 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:52.164414 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:52.165111 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:52.166573 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:52.168712 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:52.169484 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:52.169647 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:52.171091 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:52.171099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:52.171117 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:52.171777 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:52.171867 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:52.171892 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:52.172003 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:52.172350 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 16:55:52.172787 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":404,"threads_started":1}
I20260619 16:55:52.175885 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49: Bootstrap starting.
I20260619 16:55:52.178800 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:55:52.179883 21804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:52.181356 21804 log.cc:824] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:52.184785 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49: No bootstrap required, opened a new log
I20260619 16:55:52.192401 21804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3820cb27032142f2a49e08d7cb71aa49" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 38427 } }
I20260619 16:55:52.192818 21804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:52.192895 21804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3820cb27032142f2a49e08d7cb71aa49, State: Initialized, Role: FOLLOWER
I20260619 16:55:52.193440 21804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3820cb27032142f2a49e08d7cb71aa49" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 38427 } }
I20260619 16:55:52.193665 21804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:52.193765 21804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:52.193933 21804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:52.196291 21804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3820cb27032142f2a49e08d7cb71aa49" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 38427 } }
I20260619 16:55:52.196794 21804 leader_election.cc:302] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3820cb27032142f2a49e08d7cb71aa49; no voters:
I20260619 16:55:52.197382 21804 leader_election.cc:288] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:52.197548 21809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:52.198274 21809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [term 1 LEADER]: Becoming Leader. State: Replica: 3820cb27032142f2a49e08d7cb71aa49, State: Running, Role: LEADER
I20260619 16:55:52.198961 21809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3820cb27032142f2a49e08d7cb71aa49" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 38427 } }
I20260619 16:55:52.199579 21804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:52.201880 21811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3820cb27032142f2a49e08d7cb71aa49. Latest consensus state: current_term: 1 leader_uuid: "3820cb27032142f2a49e08d7cb71aa49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3820cb27032142f2a49e08d7cb71aa49" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 38427 } } }
I20260619 16:55:52.202307 21811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:52.202898 21810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3820cb27032142f2a49e08d7cb71aa49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3820cb27032142f2a49e08d7cb71aa49" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 38427 } } }
I20260619 16:55:52.203227 21810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:52.203491 21817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:52.212544 21817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:52.222908 21817 catalog_manager.cc:1347] Generated new cluster ID: b64465527d884f00b6d5cae6fc582dc0
I20260619 16:55:52.223068 21817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:52.238680 21817 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:52.240653 21817 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:52.263238 21817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3820cb27032142f2a49e08d7cb71aa49: Generated new TSK 0
I20260619 16:55:52.264464 21817 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:55:52.320874 21835 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 16:55:52.321087 21835 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:55:52.321866 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:52.291468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.16.48.62:38427)
0619 16:55:52.291727 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:52.291733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:52.292165 (+ 432us) server_negotiation.cc:407] Connection header received
0619 16:55:52.292508 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:52.292512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:52.292589 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:52.292743 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:55:52.293397 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:52.294189 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:52.296399 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:52.296892 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:52.296993 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:52.297721 (+ 728us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:55:52.321304 (+ 23583us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:55:52.321607 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":92}
W20260619 16:55:52.322032 21807 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 16:55:52.323894 21834 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:55:52.291244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:38427 (local address 127.0.0.1:45142)
0619 16:55:52.291993 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:52.292020 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:52.292269 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:52.292817 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:52.292823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:52.292840 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:55:52.293236 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:52.293248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:52.294351 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:52.294357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:52.296243 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:52.296261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:52.297090 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:52.297097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:52.297315 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:52.297411 (+ 96us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:55:52.321620 (+ 24209us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:55:52.322014 (+ 394us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.48.62:38427: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":354,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.48.62:38427: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 16:55:52.325760 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 21738
I20260619 16:55:52.327049 21799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:52.422930 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 21738
2026-06-19T16:55:52Z chronyd exiting
I20260619 16:55:52.474165 16576 test_util.cc:175] -----------------------------------------------
I20260619 16:55:52.474318 16576 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781888107067647-16576-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (797 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 16:55:52.478219 16576 mini_oidc.cc:152] Starting JWKS server
I20260619 16:55:52.480063 16576 webserver.cc:460] Webserver started at https://127.0.0.1:46783/ using document root <none> and password file <none>
I20260619 16:55:52.480376 16576 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:55:52.481070 16576 webserver.cc:460] Webserver started at http://127.0.0.1:40823/ using document root <none> and password file <none>
2026-06-19T16:55:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:52Z Disabled control of system clock
I20260619 16:55:52.510391 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:37599
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:37599
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46783/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/testchainca.cert with env {}
W20260619 16:55:52.858713 21842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:52.859094 21842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:52.859264 21842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:52.866998 21842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:52.867177 21842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:52.867269 21842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:52.867383 21842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:52.867458 21842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:55:52.867547 21842 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:55:52.867650 21842 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46783/jwks.json
I20260619 16:55:52.877838 21842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:37599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:37599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46783/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:52.879313 21842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:52.880812 21842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:52.893755 21850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:52.893843 21848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:52.893819 21847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:52.894649 21842 server_base.cc:1034] running on GCE node
I20260619 16:55:52.895561 21842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:52.897341 21842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:52.898648 21842 hybrid_clock.cc:648] HybridClock initialized: now 1781888152898598 us; error 89 us; skew 500 ppm
I20260619 16:55:52.899052 21842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:55:52.901823 21842 webserver.cc:460] Webserver started at http://127.16.48.62:36379/ using document root <none> and password file <none>
I20260619 16:55:52.902601 21842 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:52.902746 21842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:52.903095 21842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:52.906322 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "38f1687cf3be4da7aa18c52c85e04157"
format_stamp: "Formatted at 2026-06-19 16:55:52 on dist-test-slave-3ffd"
server_key: "710647c9df92babbd48b8e75f9a68214"
server_key_iv: "a32fc95c9d5bcde9b9ed03d1f56aa39f"
server_key_version: "encryptionkey@0"
I20260619 16:55:52.907244 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "38f1687cf3be4da7aa18c52c85e04157"
format_stamp: "Formatted at 2026-06-19 16:55:52 on dist-test-slave-3ffd"
server_key: "710647c9df92babbd48b8e75f9a68214"
server_key_iv: "a32fc95c9d5bcde9b9ed03d1f56aa39f"
server_key_version: "encryptionkey@0"
I20260619 16:55:52.913089 21842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:55:52.917068 21856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:52.918977 21842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:55:52.919212 21842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "38f1687cf3be4da7aa18c52c85e04157"
format_stamp: "Formatted at 2026-06-19 16:55:52 on dist-test-slave-3ffd"
server_key: "710647c9df92babbd48b8e75f9a68214"
server_key_iv: "a32fc95c9d5bcde9b9ed03d1f56aa39f"
server_key_version: "encryptionkey@0"
I20260619 16:55:52.919523 21842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:52.933887 21842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:52.934723 21842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:52.934998 21842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:52.954286 21842 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:37599
I20260619 16:55:52.954311 21907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:37599 every 8 connection(s)
I20260619 16:55:52.955821 21842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:52.959908 21908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:52.961827 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 21842
I20260619 16:55:52.962083 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:52.962968 16576 external_mini_cluster.cc:1428] Setting key 5b2c6de3f5b89091fea1a45fd38ca83e
I20260619 16:55:52.973596 21908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157: Bootstrap starting.
I20260619 16:55:52.977267 21908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:52.978137 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:52.965545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.16.48.62:37599)
0619 16:55:52.966763 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:52.966779 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:52.966827 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:55:52.968658 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:52.968691 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:52.969208 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:52.969629 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:52.970241 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:52.971599 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:52.973072 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:52.973904 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:52.974130 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:52.975445 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:52.975453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:52.975471 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:52.976052 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:52.976225 (+ 173us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:52.976255 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:52.976390 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:52.976637 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 16:55:52.976978 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":431,"threads_started":1}
I20260619 16:55:52.978498 21908 log.cc:824] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:52.981837 21908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157: No bootstrap required, opened a new log
I20260619 16:55:52.982328 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:55:52.989602 21908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f1687cf3be4da7aa18c52c85e04157" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37599 } }
I20260619 16:55:52.990069 21908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:52.990213 21908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f1687cf3be4da7aa18c52c85e04157, State: Initialized, Role: FOLLOWER
I20260619 16:55:52.990852 21908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f1687cf3be4da7aa18c52c85e04157" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37599 } }
I20260619 16:55:52.991124 21908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:52.991273 21908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:52.991461 21908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:52.993810 21908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f1687cf3be4da7aa18c52c85e04157" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37599 } }
I20260619 16:55:52.994414 21908 leader_election.cc:302] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f1687cf3be4da7aa18c52c85e04157; no voters:
I20260619 16:55:52.995055 21908 leader_election.cc:288] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:52.995229 21913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:52.995973 21913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [term 1 LEADER]: Becoming Leader. State: Replica: 38f1687cf3be4da7aa18c52c85e04157, State: Running, Role: LEADER
I20260619 16:55:52.996533 21913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f1687cf3be4da7aa18c52c85e04157" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37599 } }
I20260619 16:55:52.997339 21908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:52.999544 21914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38f1687cf3be4da7aa18c52c85e04157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f1687cf3be4da7aa18c52c85e04157" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37599 } } }
I20260619 16:55:53.000247 21914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:52.999512 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f1687cf3be4da7aa18c52c85e04157. Latest consensus state: current_term: 1 leader_uuid: "38f1687cf3be4da7aa18c52c85e04157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f1687cf3be4da7aa18c52c85e04157" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37599 } } }
I20260619 16:55:53.000782 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:53.004197 21918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:53.010105 21918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:53.019582 21918 catalog_manager.cc:1347] Generated new cluster ID: 89c6243d12f8457091b4f195c818c13a
I20260619 16:55:53.019724 21918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:53.040829 21918 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:53.042771 21918 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:53.050758 21918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38f1687cf3be4da7aa18c52c85e04157: Generated new TSK 0
I20260619 16:55:53.051915 21918 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:55:53.090636 21938 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 16:55:53.090803 21938 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:55:53.092092 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:53.070144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.16.48.62:37599)
0619 16:55:53.070363 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:53.070369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:53.070797 (+ 428us) server_negotiation.cc:407] Connection header received
0619 16:55:53.071033 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:53.071040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:53.071137 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:53.071363 (+ 226us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:55:53.071884 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.073100 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.074669 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.075414 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.075570 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:53.076238 (+ 668us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:55:53.091328 (+ 15090us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:55:53.091781 (+ 453us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 16:55:53.092192 21937 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:55:53.069984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37599 (local address 127.0.0.1:34526)
0619 16:55:53.070668 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:53.070686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:53.070826 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:53.071399 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:53.071403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:53.071413 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:55:53.071700 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:53.071711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.073300 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.073306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:53.074482 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:53.074497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.075692 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.075698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:53.075856 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:53.075897 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:55:53.091603 (+ 15706us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:55:53.091821 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.48.62:37599: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":374,"threads_started":1}
W20260619 16:55:53.092271 21911 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 16:55:53.093372 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 21842
I20260619 16:55:53.094700 21903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:53.189297 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 21842
2026-06-19T16:55:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (766 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 16:55:53.245251 16576 mini_oidc.cc:152] Starting JWKS server
I20260619 16:55:53.247206 16576 webserver.cc:460] Webserver started at https://127.0.0.1:45527/ using document root <none> and password file <none>
I20260619 16:55:53.247496 16576 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:55:53.248222 16576 webserver.cc:460] Webserver started at http://127.0.0.1:37587/ using document root <none> and password file <none>
2026-06-19T16:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:53Z Disabled control of system clock
I20260619 16:55:53.290225 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36215
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:36215
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45527/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:55:53.638705 21945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:53.639024 21945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:53.639113 21945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:53.646725 21945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:53.646858 21945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:53.646896 21945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:53.646945 21945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:53.646991 21945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:55:53.647045 21945 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:55:53.647085 21945 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45527/jwks.json
I20260619 16:55:53.656708 21945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:36215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45527/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:53.658012 21945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:53.659788 21945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:53.673616 21950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:53.673581 21951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:53.673553 21953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:53.674566 21945 server_base.cc:1034] running on GCE node
I20260619 16:55:53.675524 21945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:53.677213 21945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:53.678474 21945 hybrid_clock.cc:648] HybridClock initialized: now 1781888153678420 us; error 43 us; skew 500 ppm
I20260619 16:55:53.678942 21945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:55:53.681649 21945 webserver.cc:460] Webserver started at http://127.16.48.62:42085/ using document root <none> and password file <none>
I20260619 16:55:53.682413 21945 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:53.682534 21945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:53.682899 21945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:53.686141 21945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "e806aadbff0f468c867cc938d1c02fce"
format_stamp: "Formatted at 2026-06-19 16:55:53 on dist-test-slave-3ffd"
server_key: "7c28d3fafa76e955d231c05eb8fa1f7d"
server_key_iv: "d37c9f8d25908ecca429d445db70cb70"
server_key_version: "encryptionkey@0"
I20260619 16:55:53.687088 21945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "e806aadbff0f468c867cc938d1c02fce"
format_stamp: "Formatted at 2026-06-19 16:55:53 on dist-test-slave-3ffd"
server_key: "7c28d3fafa76e955d231c05eb8fa1f7d"
server_key_iv: "d37c9f8d25908ecca429d445db70cb70"
server_key_version: "encryptionkey@0"
I20260619 16:55:53.692965 21945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:55:53.696857 21959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:53.698354 21945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:53.698552 21945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "e806aadbff0f468c867cc938d1c02fce"
format_stamp: "Formatted at 2026-06-19 16:55:53 on dist-test-slave-3ffd"
server_key: "7c28d3fafa76e955d231c05eb8fa1f7d"
server_key_iv: "d37c9f8d25908ecca429d445db70cb70"
server_key_version: "encryptionkey@0"
I20260619 16:55:53.698796 21945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:53.714119 21945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:53.715044 21945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:53.715377 21945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:53.734297 22010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:36215 every 8 connection(s)
I20260619 16:55:53.734290 21945 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:36215
I20260619 16:55:53.735884 21945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:53.740006 22011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:53.743233 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 21945
I20260619 16:55:53.743453 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:53.744534 16576 external_mini_cluster.cc:1428] Setting key 5602f9d0d05cc37ff81bea7492d03557
I20260619 16:55:53.753407 22011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce: Bootstrap starting.
I20260619 16:55:53.756798 22011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:53.758028 22011 log.cc:824] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:53.759233 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:53.746731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.16.48.62:36215)
0619 16:55:53.747810 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:53.747822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:53.747867 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:55:53.749716 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:53.749742 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:53.750342 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:53.750814 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:53.751434 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.752830 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.754245 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.754981 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.755214 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:53.756439 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:53.756447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:53.756463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:53.757131 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:53.757241 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:53.757265 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:53.757402 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:53.757598 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 16:55:53.757929 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":372,"threads_started":1}
I20260619 16:55:53.761269 22011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce: No bootstrap required, opened a new log
I20260619 16:55:53.763347 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:55:53.769939 22011 raft_consensus.cc:348] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806aadbff0f468c867cc938d1c02fce" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36215 } }
I20260619 16:55:53.770357 22011 raft_consensus.cc:374] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:53.770478 22011 raft_consensus.cc:729] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e806aadbff0f468c867cc938d1c02fce, State: Initialized, Role: FOLLOWER
I20260619 16:55:53.771139 22011 consensus_queue.cc:260] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806aadbff0f468c867cc938d1c02fce" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36215 } }
I20260619 16:55:53.771430 22011 raft_consensus.cc:388] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:53.771580 22011 raft_consensus.cc:482] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:53.771764 22011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:53.774482 22011 raft_consensus.cc:504] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806aadbff0f468c867cc938d1c02fce" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36215 } }
I20260619 16:55:53.775070 22011 leader_election.cc:302] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e806aadbff0f468c867cc938d1c02fce; no voters:
I20260619 16:55:53.775804 22011 leader_election.cc:288] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:53.776024 22016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:53.776911 22016 raft_consensus.cc:686] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [term 1 LEADER]: Becoming Leader. State: Replica: e806aadbff0f468c867cc938d1c02fce, State: Running, Role: LEADER
I20260619 16:55:53.777563 22016 consensus_queue.cc:237] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806aadbff0f468c867cc938d1c02fce" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36215 } }
I20260619 16:55:53.778272 22011 sys_catalog.cc:564] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:53.781179 22017 sys_catalog.cc:455] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e806aadbff0f468c867cc938d1c02fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806aadbff0f468c867cc938d1c02fce" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36215 } } }
I20260619 16:55:53.781513 22017 sys_catalog.cc:458] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:53.782024 22018 sys_catalog.cc:455] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader e806aadbff0f468c867cc938d1c02fce. Latest consensus state: current_term: 1 leader_uuid: "e806aadbff0f468c867cc938d1c02fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806aadbff0f468c867cc938d1c02fce" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36215 } } }
I20260619 16:55:53.782647 22018 sys_catalog.cc:458] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:53.782650 22025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:53.791790 22025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:53.801803 22025 catalog_manager.cc:1347] Generated new cluster ID: 158511db671945f2a0a6ec5c033f83cc
I20260619 16:55:53.801934 22025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:53.825165 22025 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:53.826550 22025 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:53.833537 22025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e806aadbff0f468c867cc938d1c02fce: Generated new TSK 0
I20260619 16:55:53.834450 22025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:55:53.863634 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:53.854654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.16.48.62:36215)
0619 16:55:53.854877 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:53.854884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:53.855298 (+ 414us) server_negotiation.cc:407] Connection header received
0619 16:55:53.855509 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:53.855516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:53.855632 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:53.855831 (+ 199us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:55:53.856320 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.857503 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.858980 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.859707 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.859841 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:53.860418 (+ 577us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:55:53.862930 (+ 2512us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:55:53.863269 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 16:55:53.863767 22014 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 16:55:53.865504 22040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:55:53.854552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36215 (local address 127.0.0.1:43312)
0619 16:55:53.855198 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:53.855215 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:53.855345 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:53.855834 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:53.855837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:53.855846 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:55:53.856096 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:53.856104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.857668 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.857672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:53.858816 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:53.858830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:53.859901 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:53.859905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:53.860048 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:53.860084 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:55:53.863158 (+ 3074us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:55:53.865022 (+ 1864us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.48.62:36215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":1522,"thread_start_us":359,"threads_started":1}
I20260619 16:55:53.866442 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 21945
I20260619 16:55:53.868357 22006 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 16:55:53.875550 22041 webserver.cc:533] Webserver: sslize failed:
E20260619 16:55:53.875697 22041 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:55:53.961282 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 21945
2026-06-19T16:55:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (772 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T16:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:54Z Disabled control of system clock
I20260619 16:55:54.048255 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:35813
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:35813
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/testchainca.cert with env {}
W20260619 16:55:54.393306 22046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:54.393635 22046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:54.393731 22046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:54.401376 22046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:54.401503 22046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:54.401541 22046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:54.401589 22046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:54.401633 22046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:55:54.401705 22046 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:55:54.401744 22046 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 16:55:54.412766 22046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:35813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:35813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:54.414095 22046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:54.415397 22046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:54.428902 22054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:54.428879 22051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:54.428875 22052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:54.429616 22046 server_base.cc:1034] running on GCE node
I20260619 16:55:54.430548 22046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:54.432252 22046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:54.433514 22046 hybrid_clock.cc:648] HybridClock initialized: now 1781888154433473 us; error 58 us; skew 500 ppm
I20260619 16:55:54.433933 22046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:55:54.436744 22046 webserver.cc:460] Webserver started at http://127.16.48.62:34323/ using document root <none> and password file <none>
I20260619 16:55:54.437582 22046 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:54.437750 22046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:54.438094 22046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:54.441421 22046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "1ad9c24aca4d42719398987a8d88d3a8"
format_stamp: "Formatted at 2026-06-19 16:55:54 on dist-test-slave-3ffd"
server_key: "8cf1e51a53c354b703680108479f4ed7"
server_key_iv: "5c25c49721e26bcb870f74651535d1e9"
server_key_version: "encryptionkey@0"
I20260619 16:55:54.442368 22046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "1ad9c24aca4d42719398987a8d88d3a8"
format_stamp: "Formatted at 2026-06-19 16:55:54 on dist-test-slave-3ffd"
server_key: "8cf1e51a53c354b703680108479f4ed7"
server_key_iv: "5c25c49721e26bcb870f74651535d1e9"
server_key_version: "encryptionkey@0"
I20260619 16:55:54.448309 22046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 16:55:54.452195 22060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:54.453928 22046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:55:54.454221 22046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "1ad9c24aca4d42719398987a8d88d3a8"
format_stamp: "Formatted at 2026-06-19 16:55:54 on dist-test-slave-3ffd"
server_key: "8cf1e51a53c354b703680108479f4ed7"
server_key_iv: "5c25c49721e26bcb870f74651535d1e9"
server_key_version: "encryptionkey@0"
I20260619 16:55:54.454505 22046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:54.475046 22046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:54.476028 22046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:54.476382 22046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:54.495661 22046 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:35813
I20260619 16:55:54.495668 22111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:35813 every 8 connection(s)
I20260619 16:55:54.497225 22046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:54.500083 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 22046
I20260619 16:55:54.500300 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:54.501127 16576 external_mini_cluster.cc:1428] Setting key a6dbcf3079e97e9d29422b226db564fd
I20260619 16:55:54.502173 22112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:54.515456 22112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8: Bootstrap starting.
I20260619 16:55:54.515512 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:54.503197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.16.48.62:35813)
0619 16:55:54.504275 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:54.504288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:54.504331 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:55:54.505917 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:54.505942 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:54.506494 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:54.506967 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:54.507576 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:54.509024 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:54.510606 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:54.511514 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:54.511718 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:54.513041 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:54.513049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:54.513068 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:55:54.513692 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:54.513779 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:55:54.513805 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:54.513943 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:54.514143 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 16:55:54.514496 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":329,"threads_started":1}
I20260619 16:55:54.519537 22112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:54.519893 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:55:54.520773 22112 log.cc:824] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:54.524046 22112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8: No bootstrap required, opened a new log
I20260619 16:55:54.531809 22112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad9c24aca4d42719398987a8d88d3a8" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35813 } }
I20260619 16:55:54.532308 22112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:54.532423 22112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad9c24aca4d42719398987a8d88d3a8, State: Initialized, Role: FOLLOWER
I20260619 16:55:54.533074 22112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad9c24aca4d42719398987a8d88d3a8" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35813 } }
I20260619 16:55:54.533332 22112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:54.533483 22112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:54.533679 22112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:54.535877 22112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad9c24aca4d42719398987a8d88d3a8" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35813 } }
I20260619 16:55:54.536526 22112 leader_election.cc:302] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ad9c24aca4d42719398987a8d88d3a8; no voters:
I20260619 16:55:54.537190 22112 leader_election.cc:288] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:54.537361 22117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:54.538852 22117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [term 1 LEADER]: Becoming Leader. State: Replica: 1ad9c24aca4d42719398987a8d88d3a8, State: Running, Role: LEADER
I20260619 16:55:54.539399 22117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad9c24aca4d42719398987a8d88d3a8" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35813 } }
I20260619 16:55:54.539773 22112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:54.542585 22119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ad9c24aca4d42719398987a8d88d3a8. Latest consensus state: current_term: 1 leader_uuid: "1ad9c24aca4d42719398987a8d88d3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad9c24aca4d42719398987a8d88d3a8" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35813 } } }
I20260619 16:55:54.542551 22118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ad9c24aca4d42719398987a8d88d3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad9c24aca4d42719398987a8d88d3a8" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 35813 } } }
I20260619 16:55:54.543013 22118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:54.543013 22119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:54.547574 22126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:54.552182 22126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:54.562119 22126 catalog_manager.cc:1347] Generated new cluster ID: cad3fc50833949248aafe5ecb5e1e3f8
I20260619 16:55:54.562273 22126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:54.580789 22126 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:54.582261 22126 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:54.596848 22126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ad9c24aca4d42719398987a8d88d3a8: Generated new TSK 0
I20260619 16:55:54.597987 22126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:55:54.657785 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:54.622017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.16.48.62:35813)
0619 16:55:54.622214 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:54.622223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:54.622710 (+ 487us) server_negotiation.cc:407] Connection header received
0619 16:55:54.622969 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:54.622974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:54.623062 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:54.623209 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:55:54.623747 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:54.624578 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:54.626773 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:54.627249 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:54.627361 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:54.627971 (+ 610us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:55:54.657216 (+ 29245us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:55:54.657542 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54276: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 16:55:54.657963 22115 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54276: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 16:55:54.658646 22141 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:55:54.621749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:35813 (local address 127.0.0.1:54276)
0619 16:55:54.622573 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:54.622594 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:54.622781 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:54.623265 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:54.623270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:54.623284 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:55:54.623587 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:54.623600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:54.624748 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:54.624755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:54.626552 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:54.626569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:54.627441 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:54.627449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:54.627654 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:54.627704 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:55:54.657529 (+ 29825us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:55:54.657845 (+ 316us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.48.62:35813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":462,"threads_started":1}
I20260619 16:55:54.660221 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 22046
I20260619 16:55:54.661497 22107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:54.763947 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 22046
2026-06-19T16:55:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (781 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (432 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (430 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:55 dist-test-slave-3ffd krb5kdc[22156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:55 dist-test-slave-3ffd krb5kdc[22156](info): set up 2 sockets
Jun 19 16:55:55 dist-test-slave-3ffd krb5kdc[22156](info): commencing operation
krb5kdc: starting...
W20260619 16:55:56.757370 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:55:56 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:55:56.985661 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:34329
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:34329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:57.329936 22169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:57.330322 22169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:57.330493 22169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:57.338184 22169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:55:57.338317 22169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:57.338366 22169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:57.338415 22169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:55:57.338451 22169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:55:57.349012 22169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:34329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:57.350354 22169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:57.351679 22169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:57.364485 22174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:57.364416 22175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:57.364367 22177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:57.365207 22169 server_base.cc:1034] running on GCE node
I20260619 16:55:57.366127 22169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:57.367771 22169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:57.369004 22169 hybrid_clock.cc:648] HybridClock initialized: now 1781888157368927 us; error 79 us; skew 500 ppm
I20260619 16:55:57.369390 22169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:57 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888157, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:57.376302 22169 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:55:57.378072 22169 webserver.cc:460] Webserver started at http://127.16.48.62:42261/ using document root <none> and password file <none>
I20260619 16:55:57.378858 22169 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:57.379011 22169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:57.379350 22169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:57.382534 22169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "5974d400f9174510bfa5ae20657f3af3"
format_stamp: "Formatted at 2026-06-19 16:55:57 on dist-test-slave-3ffd"
server_key: "3587d5156816d7cda8ce2c66b033df0a"
server_key_iv: "3c57de1961420e13df8b54a4ed1690f6"
server_key_version: "encryptionkey@0"
I20260619 16:55:57.383374 22169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "5974d400f9174510bfa5ae20657f3af3"
format_stamp: "Formatted at 2026-06-19 16:55:57 on dist-test-slave-3ffd"
server_key: "3587d5156816d7cda8ce2c66b033df0a"
server_key_iv: "3c57de1961420e13df8b54a4ed1690f6"
server_key_version: "encryptionkey@0"
I20260619 16:55:57.389281 22169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:55:57.393229 22184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:57.394747 22169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:55:57.394963 22169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "5974d400f9174510bfa5ae20657f3af3"
format_stamp: "Formatted at 2026-06-19 16:55:57 on dist-test-slave-3ffd"
server_key: "3587d5156816d7cda8ce2c66b033df0a"
server_key_iv: "3c57de1961420e13df8b54a4ed1690f6"
server_key_version: "encryptionkey@0"
I20260619 16:55:57.395215 22169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:57.413792 22169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:57.418756 22169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:57.419063 22169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:57.437913 22169 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:34329
I20260619 16:55:57.437948 22235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:34329 every 8 connection(s)
I20260619 16:55:57.439442 22169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:55:57.443372 22236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:55:57.448827 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 22169
I20260619 16:55:57.449023 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:55:57.449635 16576 external_mini_cluster.cc:1428] Setting key 1fadff3f423cfde782e4064c9a19f520
I20260619 16:55:57.456408 22236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3: Bootstrap starting.
I20260619 16:55:57.460595 22236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3: Neither blocks nor log segments found. Creating new log.
I20260619 16:55:57.461824 22236 log.cc:824] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:55:57.465152 22236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3: No bootstrap required, opened a new log
Jun 19 16:55:57 dist-test-slave-3ffd krb5kdc[22156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:57.476402 22236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5974d400f9174510bfa5ae20657f3af3" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 34329 } }
I20260619 16:55:57.476866 22236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:55:57.476980 22236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5974d400f9174510bfa5ae20657f3af3, State: Initialized, Role: FOLLOWER
I20260619 16:55:57.477079 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:57.451872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.16.48.62:34329)
0619 16:55:57.452856 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:57.452876 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:57.452940 (+ 64us) server_negotiation.cc:407] Connection header received
0619 16:55:57.455477 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:57.455511 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:57.456112 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:57.456646 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:57.457997 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:57.460173 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:57.461603 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:57.462774 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:57.462926 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:57.467463 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:57.467475 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:57.467504 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:57.467567 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:57.471869 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:57.472711 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:57.472721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:57.472733 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:57.472964 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:57.473397 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:57.473402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:57.473406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:57.474257 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:57.474575 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:57.475024 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 16:55:57.475539 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":446,"threads_started":1}
I20260619 16:55:57.477665 22236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5974d400f9174510bfa5ae20657f3af3" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 34329 } }
I20260619 16:55:57.477954 22236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:55:57.478104 22236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:55:57.478300 22236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:55:57.480770 22236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5974d400f9174510bfa5ae20657f3af3" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 34329 } }
I20260619 16:55:57.481309 22236 leader_election.cc:302] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5974d400f9174510bfa5ae20657f3af3; no voters:
I20260619 16:55:57.481894 22236 leader_election.cc:288] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:55:57.482064 22241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:55:57.482779 22241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [term 1 LEADER]: Becoming Leader. State: Replica: 5974d400f9174510bfa5ae20657f3af3, State: Running, Role: LEADER
I20260619 16:55:57.483296 22241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5974d400f9174510bfa5ae20657f3af3" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 34329 } }
I20260619 16:55:57.484103 22236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:55:57.486454 22243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5974d400f9174510bfa5ae20657f3af3. Latest consensus state: current_term: 1 leader_uuid: "5974d400f9174510bfa5ae20657f3af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5974d400f9174510bfa5ae20657f3af3" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 34329 } } }
I20260619 16:55:57.486917 22243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:57.486444 22242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5974d400f9174510bfa5ae20657f3af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5974d400f9174510bfa5ae20657f3af3" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 34329 } } }
I20260619 16:55:57.487340 22242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:55:57.487811 22246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:55:57.494645 22246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:55:57.507366 22246 catalog_manager.cc:1347] Generated new cluster ID: 321180ee2cec4fd8b757d44205a4389a
I20260619 16:55:57.507508 22246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:55:57.520926 22246 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:55:57.522331 22246 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:55:57.529050 22246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5974d400f9174510bfa5ae20657f3af3: Generated new TSK 0
I20260619 16:55:57.530210 22246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:55:57.592088 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:34329
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:57.954339 22262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:57.954712 22262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:57.954883 22262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:57.962620 22262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:57.962787 22262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:57.962968 22262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:55:57.973938 22262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:57.975543 22262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:57.977068 22262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:57.992019 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:57.992208 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:57.992695 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:57.992950 22262 server_base.cc:1034] running on GCE node
I20260619 16:55:57.993638 22262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:57.994742 22262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:57.996006 22262 hybrid_clock.cc:648] HybridClock initialized: now 1781888157995944 us; error 58 us; skew 500 ppm
I20260619 16:55:57.996402 22262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:57 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888157, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:57.999770 22262 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:55:58.001461 22262 webserver.cc:460] Webserver started at http://127.16.48.1:36437/ using document root <none> and password file <none>
I20260619 16:55:58.002225 22262 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:58.002342 22262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:58.002628 22262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:58.005990 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "16db124256574b94a194fa2f63b993a8"
format_stamp: "Formatted at 2026-06-19 16:55:57 on dist-test-slave-3ffd"
server_key: "d189feebd2e6ba20fc6700ea2c32bc19"
server_key_iv: "24e45fd0b065d367a93e3f309e8ff5d1"
server_key_version: "encryptionkey@0"
I20260619 16:55:58.006891 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "16db124256574b94a194fa2f63b993a8"
format_stamp: "Formatted at 2026-06-19 16:55:57 on dist-test-slave-3ffd"
server_key: "d189feebd2e6ba20fc6700ea2c32bc19"
server_key_iv: "24e45fd0b065d367a93e3f309e8ff5d1"
server_key_version: "encryptionkey@0"
I20260619 16:55:58.012723 22262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 16:55:58.016712 22277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:58.018162 22262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:55:58.018357 22262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "16db124256574b94a194fa2f63b993a8"
format_stamp: "Formatted at 2026-06-19 16:55:57 on dist-test-slave-3ffd"
server_key: "d189feebd2e6ba20fc6700ea2c32bc19"
server_key_iv: "24e45fd0b065d367a93e3f309e8ff5d1"
server_key_version: "encryptionkey@0"
I20260619 16:55:58.018594 22262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:58.062927 22262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:58.068334 22262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:58.068668 22262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:58.069727 22262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:58.071197 22262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:58.071272 22262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:58.071421 22262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:58.071487 22262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:58.106264 22262 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:35067
I20260619 16:55:58.106317 22389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:35067 every 8 connection(s)
I20260619 16:55:58.107589 22262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:55:58.108747 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 22262
I20260619 16:55:58.108975 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:55:58.109781 16576 external_mini_cluster.cc:1428] Setting key fba3d4c1f8cc900ad64d2ac006189633
Jun 19 16:55:58 dist-test-slave-3ffd krb5kdc[22156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888157, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 16:55:58.134302 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:58.110808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:46501 (local address 127.16.48.62:34329)
0619 16:55:58.111322 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:58.111327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:58.112754 (+ 1427us) server_negotiation.cc:407] Connection header received
0619 16:55:58.115817 (+ 3063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:58.115822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:58.115898 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:58.116000 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:58.118502 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:58.119271 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:58.121109 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:58.121916 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:58.122087 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:58.126348 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:58.126358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:58.126365 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:58.126419 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:58.130326 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:58.131292 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:58.131298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:58.131302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:58.131471 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:58.132023 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:58.132028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:58.132032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:58.132475 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:58.132644 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:58.133618 (+ 974us) server_negotiation.cc:299] Negotiation successful
0619 16:55:58.133886 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":261,"threads_started":1}
I20260619 16:55:58.135349 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:58.111531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:34329 (local address 127.16.48.1:46501)
0619 16:55:58.112556 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:58.112615 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:58.115427 (+ 2812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:58.116362 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:58.116378 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:58.117269 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:58.118230 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:58.118263 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:58.119518 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:58.119522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:58.120936 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:58.120950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:58.122198 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:58.122208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:58.122462 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:58.123266 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:58.123302 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:58.125993 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:58.130556 (+ 4563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:58.130583 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:58.130600 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:58.131114 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:58.131630 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:58.131635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:58.131637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:58.131870 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:58.132650 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:58.132660 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:58.133155 (+ 495us) client_negotiation.cc:769] Sending connection context
0619 16:55:58.133433 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 16:55:58.133962 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":444,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:55:58.137266 22390 heartbeater.cc:347] Connected to a master server at 127.16.48.62:34329
I20260619 16:55:58.137625 22390 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:58.138413 22390 heartbeater.cc:511] Master 127.16.48.62:34329 requested a full tablet report, sending...
I20260619 16:55:58.140784 22201 ts_manager.cc:194] Registered new tserver with Master: 16db124256574b94a194fa2f63b993a8 (127.16.48.1:35067)
I20260619 16:55:58.142514 22201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:46501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:55:58.164206 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:34329
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:58.509620 22396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:58.509920 22396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:58.510095 22396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:58.517810 22396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:58.517946 22396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:58.518141 22396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:55:58.528801 22396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:58.530207 22396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:58.531572 22396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:58.545554 22404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:58.545643 22402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:58.545595 22401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:58.547032 22396 server_base.cc:1034] running on GCE node
I20260619 16:55:58.547757 22396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:58.548866 22396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:58.550117 22396 hybrid_clock.cc:648] HybridClock initialized: now 1781888158550065 us; error 49 us; skew 500 ppm
I20260619 16:55:58.550477 22396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:58 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888158, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:58.553763 22396 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:55:58.555369 22396 webserver.cc:460] Webserver started at http://127.16.48.2:36975/ using document root <none> and password file <none>
I20260619 16:55:58.556066 22396 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:58.556246 22396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:58.556536 22396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:58.559535 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "01ab9d788dca401d987b3234ed741a29"
format_stamp: "Formatted at 2026-06-19 16:55:58 on dist-test-slave-3ffd"
server_key: "09b9fbae742e76d7d14803c3c34881bd"
server_key_iv: "b51e07bbb6ff293b4855791f6fadc6fa"
server_key_version: "encryptionkey@0"
I20260619 16:55:58.560434 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "01ab9d788dca401d987b3234ed741a29"
format_stamp: "Formatted at 2026-06-19 16:55:58 on dist-test-slave-3ffd"
server_key: "09b9fbae742e76d7d14803c3c34881bd"
server_key_iv: "b51e07bbb6ff293b4855791f6fadc6fa"
server_key_version: "encryptionkey@0"
I20260619 16:55:58.566241 22396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:55:58.570109 22411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:58.571591 22396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:55:58.571796 22396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "01ab9d788dca401d987b3234ed741a29"
format_stamp: "Formatted at 2026-06-19 16:55:58 on dist-test-slave-3ffd"
server_key: "09b9fbae742e76d7d14803c3c34881bd"
server_key_iv: "b51e07bbb6ff293b4855791f6fadc6fa"
server_key_version: "encryptionkey@0"
I20260619 16:55:58.572012 22396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:58.584383 22396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:58.589119 22396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:58.589500 22396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:58.590574 22396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:58.591989 22396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:58.592097 22396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:58.592314 22396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:58.592391 22396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:58.627310 22396 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:35579
I20260619 16:55:58.627353 22523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:35579 every 8 connection(s)
I20260619 16:55:58.628731 22396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:55:58.636597 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 22396
I20260619 16:55:58.636781 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:55:58.637245 16576 external_mini_cluster.cc:1428] Setting key 2393d1845e045cfdfb6229e9e962ab97
Jun 19 16:55:58 dist-test-slave-3ffd krb5kdc[22156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888158, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:58.650566 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:58.631331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:54981 (local address 127.16.48.62:34329)
0619 16:55:58.631509 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:58.631513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:58.632913 (+ 1400us) server_negotiation.cc:407] Connection header received
0619 16:55:58.634815 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:58.634821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:58.634920 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:58.635034 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:58.636959 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:58.637743 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:58.639608 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:58.640028 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:58.640196 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:58.644012 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:58.644017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:58.644020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:58.644053 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:58.646969 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:58.648047 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:58.648050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:58.648052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:58.648232 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:58.648771 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:58.648774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:58.648776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:58.649054 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:58.649190 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:58.650144 (+ 954us) server_negotiation.cc:299] Negotiation successful
0619 16:55:58.650331 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:55:58.652401 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:58.631880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:34329 (local address 127.16.48.2:54981)
0619 16:55:58.632723 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:58.632764 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:58.634494 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:58.635295 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:58.635304 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:58.635860 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:58.636727 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:58.636761 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:58.637910 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:58.637918 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:58.639457 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:58.639475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:58.640281 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:58.640288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:58.640526 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:58.641242 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:58.641266 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:58.643746 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:58.647159 (+ 3413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:58.647174 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:58.647205 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:58.647903 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:58.648354 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:58.648362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:58.648369 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:58.648635 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:58.649211 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:58.649223 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:58.649720 (+ 497us) client_negotiation.cc:769] Sending connection context
0619 16:55:58.650063 (+ 343us) client_negotiation.cc:240] Negotiation successful
0619 16:55:58.650721 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":381,"threads_started":1}
I20260619 16:55:58.654070 22524 heartbeater.cc:347] Connected to a master server at 127.16.48.62:34329
I20260619 16:55:58.654428 22524 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:58.655182 22524 heartbeater.cc:511] Master 127.16.48.62:34329 requested a full tablet report, sending...
I20260619 16:55:58.657006 22201 ts_manager.cc:194] Registered new tserver with Master: 01ab9d788dca401d987b3234ed741a29 (127.16.48.2:35579)
I20260619 16:55:58.657866 22201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:54981
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:55:58.688444 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:34329
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:55:59.038564 22529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:55:59.038919 22529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:55:59.039098 22529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:55:59.046648 22529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:55:59.046833 22529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:55:59.047021 22529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:55:59.058516 22529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:55:59.059929 22529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:55:59.061408 22529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:55:59.075711 22537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:59.075662 22535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:55:59.076371 22534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:55:59.076373 22529 server_base.cc:1034] running on GCE node
I20260619 16:55:59.077219 22529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:55:59.078323 22529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:55:59.079610 22529 hybrid_clock.cc:648] HybridClock initialized: now 1781888159079550 us; error 56 us; skew 500 ppm
I20260619 16:55:59.080013 22529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:59.083515 22529 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:55:59.085357 22529 webserver.cc:460] Webserver started at http://127.16.48.3:33405/ using document root <none> and password file <none>
I20260619 16:55:59.086128 22529 fs_manager.cc:359] Metadata directory not provided
I20260619 16:55:59.086282 22529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:55:59.086615 22529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:55:59.089876 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "3213442b756543bfbf168101271d8626"
format_stamp: "Formatted at 2026-06-19 16:55:59 on dist-test-slave-3ffd"
server_key: "eaf4544287d191ed8884cd6a97eec2d6"
server_key_iv: "d8f1fb92571a9110f888c119d8deb90f"
server_key_version: "encryptionkey@0"
I20260619 16:55:59.090700 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "3213442b756543bfbf168101271d8626"
format_stamp: "Formatted at 2026-06-19 16:55:59 on dist-test-slave-3ffd"
server_key: "eaf4544287d191ed8884cd6a97eec2d6"
server_key_iv: "d8f1fb92571a9110f888c119d8deb90f"
server_key_version: "encryptionkey@0"
I20260619 16:55:59.096316 22529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:55:59.101363 22544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:59.103202 22529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:55:59.103420 22529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "3213442b756543bfbf168101271d8626"
format_stamp: "Formatted at 2026-06-19 16:55:59 on dist-test-slave-3ffd"
server_key: "eaf4544287d191ed8884cd6a97eec2d6"
server_key_iv: "d8f1fb92571a9110f888c119d8deb90f"
server_key_version: "encryptionkey@0"
I20260619 16:55:59.103665 22529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:55:59.128963 22529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:55:59.133591 22529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:55:59.133915 22529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:55:59.134953 22529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:55:59.136382 22529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:55:59.136494 22529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:59.136670 22529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:55:59.136750 22529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:55:59.146219 22390 heartbeater.cc:503] Master 127.16.48.62:34329 was elected leader, sending a full tablet report...
I20260619 16:55:59.171159 22529 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:37977
I20260619 16:55:59.171255 22656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:37977 every 8 connection(s)
I20260619 16:55:59.172655 22529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:55:59.173986 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 22529
I20260619 16:55:59.174170 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:55:59.174760 16576 external_mini_cluster.cc:1428] Setting key c0de7e68adfbbbc7a2aee740bdc4e8fc
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:55:59.195685 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:59.175754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:54707 (local address 127.16.48.62:34329)
0619 16:55:59.176307 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.176312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.177855 (+ 1543us) server_negotiation.cc:407] Connection header received
0619 16:55:59.180007 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.180012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.180106 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.180275 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.181990 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.182863 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.184804 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.185346 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.185513 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:59.189360 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:59.189366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:59.189371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:59.189413 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:59.192735 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.193552 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.193556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.193558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.193702 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.194106 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.194109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.194111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.194373 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:59.194491 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:59.195224 (+ 733us) server_negotiation.cc:299] Negotiation successful
0619 16:55:59.195402 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":265,"threads_started":1}
I20260619 16:55:59.196892 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:59.176612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:34329 (local address 127.16.48.3:54707)
0619 16:55:59.177663 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0619 16:55:59.177728 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:55:59.179678 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:55:59.180561 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:55:59.180570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:55:59.181148 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:55:59.181786 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:59.181807 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.183047 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.183060 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:59.184629 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:55:59.184645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.185613 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.185621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:55:59.185890 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:59.186605 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:55:59.186630 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:55:59.189102 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:55:59.192895 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:59.192904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:59.192920 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:59.193423 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:59.193799 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:55:59.193802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:55:59.193804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:55:59.194003 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:55:59.194494 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:55:59.194500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:55:59.194873 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 16:55:59.195120 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 16:55:59.195549 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":419,"threads_started":1}
I20260619 16:55:59.198505 22657 heartbeater.cc:347] Connected to a master server at 127.16.48.62:34329
I20260619 16:55:59.198853 22657 heartbeater.cc:464] Registering TS with master...
I20260619 16:55:59.199698 22657 heartbeater.cc:511] Master 127.16.48.62:34329 requested a full tablet report, sending...
I20260619 16:55:59.201414 22201 ts_manager.cc:194] Registered new tserver with Master: 3213442b756543bfbf168101271d8626 (127.16.48.3:37977)
I20260619 16:55:59.202312 22201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:54707
I20260619 16:55:59.203994 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/test-admin.keytab.
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:59.233206 16576 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 16:55:59.238756 22665 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:55:59.235946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.16.48.1:35067)
0619 16:55:59.236621 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.236629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.236668 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:55:59.236893 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.236900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.237133 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.237290 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.238337 (+ 1047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53630: BlockingRecv error: recv got EOF from 127.0.0.1:53630 (error 108)
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":373,"threads_started":1}
W20260619 16:55:59.238848 16576 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:59.240846 16576 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:59.242333 16576 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:59.259016 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:59.244321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.16.48.1:35067)
0619 16:55:59.244532 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.244545 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.244661 (+ 116us) server_negotiation.cc:407] Connection header received
0619 16:55:59.244826 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.244830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.244891 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.244978 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.246040 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.247041 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.248324 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.248907 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.249012 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:59.252927 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:59.252942 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:59.252959 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:59.253006 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:59.255822 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.256668 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.256677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.256680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.256857 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.257324 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.257329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.257332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.257796 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:59.258116 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:59.258438 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 16:55:59.258714 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 16:55:59.265311 22665 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:55:59.263416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.16.48.1:35067)
0619 16:55:59.263576 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.263580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.263997 (+ 417us) server_negotiation.cc:407] Connection header received
0619 16:55:59.264184 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.264188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.264251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.264332 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.265028 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53656: BlockingRecv error: recv got EOF from 127.0.0.1:53656 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 16:55:59.265478 16576 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:59.267009 16576 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:59.268399 16576 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:59.283034 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:59.269914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.16.48.1:35067)
0619 16:55:59.270079 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.270084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.270459 (+ 375us) server_negotiation.cc:407] Connection header received
0619 16:55:59.270593 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.270597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.270678 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.270763 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.271747 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.272524 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.273718 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.274124 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.274231 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:59.278090 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:59.278095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:59.278098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:59.278154 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:59.280672 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.281337 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.281341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.281344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.281464 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.281794 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.281797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.281799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.282052 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:59.282270 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:59.282624 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 16:55:59.282813 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 16:55:59.287585 22665 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:55:59.285917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.16.48.1:35067)
0619 16:55:59.286118 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.286122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.286238 (+ 116us) server_negotiation.cc:407] Connection header received
0619 16:55:59.286371 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.286375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.286447 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.286524 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.287341 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53676: BlockingRecv error: recv got EOF from 127.0.0.1:53676 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 16:55:59.287736 16576 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781888107067647-16576-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:59.289408 16576 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:55:59.290791 16576 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:55:59.305733 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:59.292252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.16.48.1:35067)
0619 16:55:59.292452 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.292462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.292787 (+ 325us) server_negotiation.cc:407] Connection header received
0619 16:55:59.292992 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.292996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.293059 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.293167 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.294145 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.294863 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.296102 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.296561 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.296714 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:59.300461 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:59.300466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:59.300470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:59.300510 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:59.302970 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.303706 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.303712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.303716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.303864 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.304316 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.304321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.304325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.304616 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:59.304959 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:59.305176 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 16:55:59.305405 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:55:59.307322 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 22262
I20260619 16:55:59.320341 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:59.308245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.16.48.1:35067)
0619 16:55:59.308442 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.308447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.308620 (+ 173us) server_negotiation.cc:407] Connection header received
0619 16:55:59.308821 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.308825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.308879 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.308956 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.309954 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.310619 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.311779 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.312274 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.312367 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:59.314614 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:59.314618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:59.314622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:59.314656 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:59.317521 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.318340 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.318344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.318347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.318538 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.318891 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.318895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.318897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.319183 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:59.319397 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:59.319655 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 16:55:59.319893 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:55:59.321031 22385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:59.405478 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 22262
I20260619 16:55:59.422261 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 22396
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:55:59.438059 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:59.423202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.16.48.2:35579)
0619 16:55:59.423743 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.423749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.423775 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:55:59.423832 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.423839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.424054 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.424268 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.425300 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.426316 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.427589 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.428253 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.428399 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:59.432222 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:59.432236 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:59.432253 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:59.432289 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:59.435041 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.435788 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.435796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.435798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.435925 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.436406 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.436411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.436413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.436757 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:59.437072 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:59.437417 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 16:55:59.437691 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":267,"threads_started":1}
I20260619 16:55:59.439433 22519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:59.524540 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 22396
I20260619 16:55:59.540786 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 22529
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:55:59.557379 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:55:59.541785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52542 (local address 127.16.48.3:37977)
0619 16:55:59.542427 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 16:55:59.542434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:55:59.542466 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:55:59.542527 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:55:59.542533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:55:59.542756 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:55:59.542937 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:55:59.544201 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.545239 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.546502 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:55:59.547091 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:55:59.547255 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:55:59.551375 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:55:59.551391 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:55:59.551408 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:55:59.551454 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:55:59.554533 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.555376 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.555384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.555386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.555518 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:55:59.555893 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:55:59.555896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:55:59.555898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:55:59.556349 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:55:59.556581 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 16:55:59.556882 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 16:55:59.557091 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":336,"threads_started":1}
I20260619 16:55:59.558677 22652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:59.644008 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 22529
I20260619 16:55:59.659948 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 22169
I20260619 16:55:59.661233 22231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:55:59.777853 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 22169
2026-06-19T16:55:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4156 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22685](info): set up 2 sockets
Jun 19 16:55:59 dist-test-slave-3ffd krb5kdc[22685](info): commencing operation
krb5kdc: starting...
W20260619 16:56:00.893234 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:56:01 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:56:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.62.keytab.
Entry for principal oryx/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.62.keytab.
I20260619 16:56:01.126156 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:41377
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.62.keytab
--principal=oryx/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:41377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:01.474542 22698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:01.474865 22698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:01.475051 22698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:01.482609 22698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:01.482739 22698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:01.482861 22698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:01.482965 22698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:01.483049 22698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:01.493666 22698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:41377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.62.keytab
--principal=oryx/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:41377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:01.495111 22698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:01.496647 22698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:01.509616 22704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:01.509655 22703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:01.509580 22706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:01.510506 22698 server_base.cc:1034] running on GCE node
I20260619 16:56:01.511464 22698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:01.513141 22698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:01.514396 22698 hybrid_clock.cc:648] HybridClock initialized: now 1781888161514349 us; error 41 us; skew 500 ppm
I20260619 16:56:01.514840 22698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:01 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:01.518452 22698 init.cc:377] Logged in from keytab as oryx/127.16.48.62@KRBTEST.COM (short username oryx)
I20260619 16:56:01.520486 22698 webserver.cc:460] Webserver started at http://127.16.48.62:33109/ using document root <none> and password file <none>
I20260619 16:56:01.521274 22698 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:01.521440 22698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:01.521778 22698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:01.525172 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "7789317184b442768fe9397c633f7ee6"
format_stamp: "Formatted at 2026-06-19 16:56:01 on dist-test-slave-3ffd"
server_key: "c7e27b7617dd014f12b3b0c6c9261db2"
server_key_iv: "0d13937a3f77d97d99f8bd7eb35bbb97"
server_key_version: "encryptionkey@0"
I20260619 16:56:01.526103 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "7789317184b442768fe9397c633f7ee6"
format_stamp: "Formatted at 2026-06-19 16:56:01 on dist-test-slave-3ffd"
server_key: "c7e27b7617dd014f12b3b0c6c9261db2"
server_key_iv: "0d13937a3f77d97d99f8bd7eb35bbb97"
server_key_version: "encryptionkey@0"
I20260619 16:56:01.531993 22698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:56:01.535805 22713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:01.537487 22698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:56:01.537712 22698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "7789317184b442768fe9397c633f7ee6"
format_stamp: "Formatted at 2026-06-19 16:56:01 on dist-test-slave-3ffd"
server_key: "c7e27b7617dd014f12b3b0c6c9261db2"
server_key_iv: "0d13937a3f77d97d99f8bd7eb35bbb97"
server_key_version: "encryptionkey@0"
I20260619 16:56:01.537940 22698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:01.556618 22698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:01.561352 22698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:01.561684 22698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:01.581933 22698 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:41377
I20260619 16:56:01.581954 22764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:41377 every 8 connection(s)
I20260619 16:56:01.583513 22698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:56:01.587642 22765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:01.589531 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 22698
I20260619 16:56:01.589751 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:56:01.590400 16576 external_mini_cluster.cc:1428] Setting key edc8515c3df72b6538999aece30c3798
I20260619 16:56:01.602015 22765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: Bootstrap starting.
I20260619 16:56:01.605860 22765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: Neither blocks nor log segments found. Creating new log.
Jun 19 16:56:01 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:01.607080 22765 log.cc:824] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:01.611001 22765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: No bootstrap required, opened a new log
I20260619 16:56:01.617298 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:01.592648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.16.48.62:41377)
0619 16:56:01.593518 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:01.593536 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:01.593595 (+ 59us) server_negotiation.cc:407] Connection header received
0619 16:56:01.596011 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:01.596042 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:01.596633 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:01.597124 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:01.598529 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:01.600771 (+ 2242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:01.602251 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:01.603306 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:01.603526 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:01.607921 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:01.607930 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:01.607947 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:01.607992 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:01.612003 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:01.612891 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:01.612901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:01.612914 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:01.613158 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:01.613552 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:01.613558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:01.613562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:01.614378 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:01.614737 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:01.615195 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 16:56:01.615720 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":364,"threads_started":1}
I20260619 16:56:01.621346 22765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } }
I20260619 16:56:01.621758 22765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:01.621833 22765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7789317184b442768fe9397c633f7ee6, State: Initialized, Role: FOLLOWER
I20260619 16:56:01.622359 22765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } }
I20260619 16:56:01.622570 22765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:56:01.622670 22765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:56:01.622797 22765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:01.625124 22765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } }
I20260619 16:56:01.625660 22765 leader_election.cc:302] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7789317184b442768fe9397c633f7ee6; no voters:
I20260619 16:56:01.626272 22765 leader_election.cc:288] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:56:01.626397 22770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:01.627846 22770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 1 LEADER]: Becoming Leader. State: Replica: 7789317184b442768fe9397c633f7ee6, State: Running, Role: LEADER
I20260619 16:56:01.628499 22770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } }
I20260619 16:56:01.628876 22765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:56:01.632349 22772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7789317184b442768fe9397c633f7ee6. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } }
I20260619 16:56:01.632334 22771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } }
I20260619 16:56:01.632781 22772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:01.632838 22771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:01.635083 22778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:56:01.640935 22778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:56:01.650957 22778 catalog_manager.cc:1347] Generated new cluster ID: b3fe076130634d9a832f2c816f6e384e
I20260619 16:56:01.651108 22778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:56:01.672843 22778 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:56:01.674896 22778 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:56:01.690267 22778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: Generated new TSK 0
I20260619 16:56:01.691469 22778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab.
Entry for principal oryx/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab.
I20260619 16:56:01.764411 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:41377
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab
--principal=oryx/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:02.126791 22791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:02.127193 22791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:02.127363 22791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:02.135564 22791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:02.135725 22791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:02.135912 22791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:56:02.146682 22791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab
--principal=oryx/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:41377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:02.148177 22791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:02.149586 22791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:02.163659 22799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:02.163687 22797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:02.163722 22796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:02.164451 22791 server_base.cc:1034] running on GCE node
I20260619 16:56:02.165306 22791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:02.166441 22791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:02.167721 22791 hybrid_clock.cc:648] HybridClock initialized: now 1781888162167659 us; error 55 us; skew 500 ppm
I20260619 16:56:02.168098 22791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:02 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888162, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:02.171579 22791 init.cc:377] Logged in from keytab as oryx/127.16.48.1@KRBTEST.COM (short username oryx)
I20260619 16:56:02.173237 22791 webserver.cc:460] Webserver started at http://127.16.48.1:35799/ using document root <none> and password file <none>
I20260619 16:56:02.173945 22791 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:02.174053 22791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:02.174314 22791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:02.177541 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "7552fdd23713435ab989a0150f76e121"
format_stamp: "Formatted at 2026-06-19 16:56:02 on dist-test-slave-3ffd"
server_key: "ef37236acc2434bac03e913ae76f32b1"
server_key_iv: "ca3b05677aa9f9805d9fa57202197880"
server_key_version: "encryptionkey@0"
I20260619 16:56:02.178351 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "7552fdd23713435ab989a0150f76e121"
format_stamp: "Formatted at 2026-06-19 16:56:02 on dist-test-slave-3ffd"
server_key: "ef37236acc2434bac03e913ae76f32b1"
server_key_iv: "ca3b05677aa9f9805d9fa57202197880"
server_key_version: "encryptionkey@0"
I20260619 16:56:02.184350 22791 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 16:56:02.188877 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:02.190703 22791 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:56:02.190919 22791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "7552fdd23713435ab989a0150f76e121"
format_stamp: "Formatted at 2026-06-19 16:56:02 on dist-test-slave-3ffd"
server_key: "ef37236acc2434bac03e913ae76f32b1"
server_key_iv: "ca3b05677aa9f9805d9fa57202197880"
server_key_version: "encryptionkey@0"
I20260619 16:56:02.191155 22791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:02.220046 22791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:02.224963 22791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:02.225302 22791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:02.226362 22791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:02.227772 22791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:02.227846 22791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:02.228015 22791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:02.228083 22791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:02.262471 22791 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:39311
I20260619 16:56:02.262521 22918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:39311 every 8 connection(s)
I20260619 16:56:02.263826 22791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:56:02.270560 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 22791
I20260619 16:56:02.270772 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:56:02.271456 16576 external_mini_cluster.cc:1428] Setting key c51d0940e60e1e90ea14bb10cd45189b
Jun 19 16:56:02 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888162, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:02.285826 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:02.266467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:39301 (local address 127.16.48.62:41377)
0619 16:56:02.266917 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:02.266923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:02.267937 (+ 1014us) server_negotiation.cc:407] Connection header received
0619 16:56:02.269723 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:02.269727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:02.269792 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:02.269887 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:02.271491 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:02.272479 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:02.274646 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:02.275054 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:02.275168 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:02.279800 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:02.279805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:02.279808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:02.279844 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:02.282541 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:02.283347 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:02.283354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:02.283358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:02.283486 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:02.283938 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:02.283946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:02.283950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:02.284293 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:02.284448 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:02.285221 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 16:56:02.285452 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":211,"threads_started":1}
I20260619 16:56:02.286926 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:02.267068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.1:39301)
0619 16:56:02.267775 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:02.267813 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:02.269473 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:02.270100 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:02.270108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:02.270614 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:02.271305 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:02.271323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:02.272699 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:02.272703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:02.274441 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:02.274462 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:02.275330 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:02.275341 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:02.275735 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:02.276596 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:02.276633 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:02.279473 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:02.282720 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:02.282731 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:02.282751 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:02.283214 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:02.283603 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:02.283609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:02.283612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:02.283821 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:02.284421 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:02.284431 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:02.284802 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 16:56:02.285081 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 16:56:02.285697 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":272,"threads_started":1}
I20260619 16:56:02.288625 22919 heartbeater.cc:347] Connected to a master server at 127.16.48.62:41377
I20260619 16:56:02.289024 22919 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:02.290062 22919 heartbeater.cc:511] Master 127.16.48.62:41377 requested a full tablet report, sending...
I20260619 16:56:02.292659 22730 ts_manager.cc:194] Registered new tserver with Master: 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311)
WARNING: no policy specified for oryx/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 16:56:02.294234 22730 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:39301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab.
Entry for principal oryx/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab.
I20260619 16:56:02.323361 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:41377
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab
--principal=oryx/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:02.676795 22925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:02.677111 22925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:02.677253 22925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:02.685092 22925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:02.685247 22925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:02.685488 22925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:56:02.696728 22925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab
--principal=oryx/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:41377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:02.698612 22925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:02.700654 22925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:02.715344 22930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:02.715262 22933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:02.715322 22931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:02.716758 22925 server_base.cc:1034] running on GCE node
I20260619 16:56:02.717480 22925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:02.718648 22925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:02.719928 22925 hybrid_clock.cc:648] HybridClock initialized: now 1781888162719863 us; error 71 us; skew 500 ppm
I20260619 16:56:02.720363 22925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:02 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888162, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:02.727312 22925 init.cc:377] Logged in from keytab as oryx/127.16.48.2@KRBTEST.COM (short username oryx)
I20260619 16:56:02.729209 22925 webserver.cc:460] Webserver started at http://127.16.48.2:39311/ using document root <none> and password file <none>
I20260619 16:56:02.730059 22925 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:02.730217 22925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:02.730547 22925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:02.733731 22925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "731e6bc712914e86a3c5307622e64f51"
format_stamp: "Formatted at 2026-06-19 16:56:02 on dist-test-slave-3ffd"
server_key: "69313e8da11e71c4befb8844f6c13a62"
server_key_iv: "adbcb75c2b8e7c187f086d246a7cf2a8"
server_key_version: "encryptionkey@0"
I20260619 16:56:02.734633 22925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "731e6bc712914e86a3c5307622e64f51"
format_stamp: "Formatted at 2026-06-19 16:56:02 on dist-test-slave-3ffd"
server_key: "69313e8da11e71c4befb8844f6c13a62"
server_key_iv: "adbcb75c2b8e7c187f086d246a7cf2a8"
server_key_version: "encryptionkey@0"
I20260619 16:56:02.740357 22925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:56:02.744454 22940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:02.746116 22925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:02.746336 22925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "731e6bc712914e86a3c5307622e64f51"
format_stamp: "Formatted at 2026-06-19 16:56:02 on dist-test-slave-3ffd"
server_key: "69313e8da11e71c4befb8844f6c13a62"
server_key_iv: "adbcb75c2b8e7c187f086d246a7cf2a8"
server_key_version: "encryptionkey@0"
I20260619 16:56:02.746578 22925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:02.768926 22925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:02.773383 22925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:02.773707 22925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:02.774807 22925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:02.776296 22925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:02.776408 22925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:02.776562 22925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:02.776633 22925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:02.810148 22925 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:40595
I20260619 16:56:02.810238 23052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:40595 every 8 connection(s)
I20260619 16:56:02.811558 22925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:56:02.816700 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 22925
I20260619 16:56:02.816902 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:56:02.817358 16576 external_mini_cluster.cc:1428] Setting key 431b14a78b345bee94d1a26edceb1048
Jun 19 16:56:02 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888162, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:02.834339 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:02.814364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:49681 (local address 127.16.48.62:41377)
0619 16:56:02.814803 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:02.814808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:02.815773 (+ 965us) server_negotiation.cc:407] Connection header received
0619 16:56:02.818041 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:02.818046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:02.818156 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:02.818286 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:02.820376 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:02.821108 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:02.822823 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:02.823252 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:02.823399 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:02.827652 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:02.827684 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:02.827688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:02.827724 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:02.830616 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:02.831485 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:02.831490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:02.831493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:02.831620 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:02.832094 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:02.832103 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:02.832105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:02.832438 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:02.832599 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:02.833471 (+ 872us) server_negotiation.cc:299] Negotiation successful
0619 16:56:02.833776 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":231,"threads_started":1}
I20260619 16:56:02.835052 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:02.814864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:49681)
0619 16:56:02.815589 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:02.815635 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:02.817678 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:02.818609 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:02.818622 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:02.819474 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:02.820190 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:02.820214 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:02.821278 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:02.821283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:02.822641 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:02.822662 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:02.823513 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:02.823520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:02.823777 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:02.824574 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:02.824604 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:02.827361 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:02.830812 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:02.830823 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:02.830851 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:02.831327 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:02.831750 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:02.831756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:02.831760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:02.831966 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:02.832574 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:02.832584 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:02.832971 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 16:56:02.833271 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 16:56:02.833784 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":279,"threads_started":1}
I20260619 16:56:02.836862 23053 heartbeater.cc:347] Connected to a master server at 127.16.48.62:41377
I20260619 16:56:02.837194 23053 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.16.48.3@KRBTEST.COM; defaulting to no policy
I20260619 16:56:02.838011 23053 heartbeater.cc:511] Master 127.16.48.62:41377 requested a full tablet report, sending...
I20260619 16:56:02.839936 22730 ts_manager.cc:194] Registered new tserver with Master: 731e6bc712914e86a3c5307622e64f51 (127.16.48.2:40595)
I20260619 16:56:02.840865 22730 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:49681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab.
Entry for principal oryx/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab.
I20260619 16:56:02.868939 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:41377
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab
--principal=oryx/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:03.217645 23059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:03.217967 23059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:03.218063 23059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:03.225534 23059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:03.225667 23059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:03.225793 23059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:56:03.236073 23059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab
--principal=oryx/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:41377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:03.237435 23059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:03.238754 23059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:03.253522 23067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:03.253578 23065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:03.253764 23064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:03.254873 23059 server_base.cc:1034] running on GCE node
I20260619 16:56:03.255563 23059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:03.256752 23059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:03.257992 23059 hybrid_clock.cc:648] HybridClock initialized: now 1781888163257937 us; error 55 us; skew 500 ppm
I20260619 16:56:03.258337 23059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:03 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888163, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:56:03.267027 22915 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.16.48.1:39311
I20260619 16:56:03.267568 23059 init.cc:377] Logged in from keytab as oryx/127.16.48.3@KRBTEST.COM (short username oryx)
I20260619 16:56:03.269259 23059 webserver.cc:460] Webserver started at http://127.16.48.3:37847/ using document root <none> and password file <none>
I20260619 16:56:03.270026 23059 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:03.270182 23059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:03.270520 23059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:03.273713 23059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "2d29389ddcd3472c9b6ea7e1cba2078e"
format_stamp: "Formatted at 2026-06-19 16:56:03 on dist-test-slave-3ffd"
server_key: "829a70a80e90dc3aedb715cf59602005"
server_key_iv: "d2cadca786efa38fdaf00b857e37655c"
server_key_version: "encryptionkey@0"
I20260619 16:56:03.274556 23059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "2d29389ddcd3472c9b6ea7e1cba2078e"
format_stamp: "Formatted at 2026-06-19 16:56:03 on dist-test-slave-3ffd"
server_key: "829a70a80e90dc3aedb715cf59602005"
server_key_iv: "d2cadca786efa38fdaf00b857e37655c"
server_key_version: "encryptionkey@0"
I20260619 16:56:03.280457 23059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:56:03.284397 23074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:03.285969 23059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:03.286201 23059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "2d29389ddcd3472c9b6ea7e1cba2078e"
format_stamp: "Formatted at 2026-06-19 16:56:03 on dist-test-slave-3ffd"
server_key: "829a70a80e90dc3aedb715cf59602005"
server_key_iv: "d2cadca786efa38fdaf00b857e37655c"
server_key_version: "encryptionkey@0"
I20260619 16:56:03.286504 23059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:03.297699 22919 heartbeater.cc:503] Master 127.16.48.62:41377 was elected leader, sending a full tablet report...
I20260619 16:56:03.301278 23059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:03.306365 23059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:03.306735 23059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:03.307998 23059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:03.310839 23059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:03.310966 23059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:03.311113 23059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:03.311184 23059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:03.357087 23059 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:33621
I20260619 16:56:03.357121 23186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:33621 every 8 connection(s)
I20260619 16:56:03.359154 23059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:56:03.362859 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 23059
I20260619 16:56:03.363047 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:56:03.363619 16576 external_mini_cluster.cc:1428] Setting key a8b05a8224baf610c79d3fe5734a0a2f
Jun 19 16:56:03 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888163, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:03.385368 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:03.363693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:49077 (local address 127.16.48.62:41377)
0619 16:56:03.364205 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:03.364212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:03.367481 (+ 3269us) server_negotiation.cc:407] Connection header received
0619 16:56:03.367554 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:03.367561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:03.367699 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:03.367826 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:03.370410 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.371844 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.374257 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.374689 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.374832 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:03.378971 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:03.378976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:03.378979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:03.379017 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:03.381908 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:03.382799 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:03.382804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:03.382806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:03.382928 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:03.383396 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:03.383409 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:03.383411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:03.383694 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:03.383861 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:03.384714 (+ 853us) server_negotiation.cc:299] Negotiation successful
0619 16:56:03.384955 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":1116,"threads_started":1}
I20260619 16:56:03.386404 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:03.363137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.3:49077)
0619 16:56:03.364281 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:03.364337 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:03.367004 (+ 2667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:03.368252 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:03.368266 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:03.369190 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:03.370109 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:03.370139 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.372070 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.372074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:03.374044 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:03.374061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.374970 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.374976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:03.375219 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:03.375974 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:03.375998 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:03.378709 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:03.382107 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:03.382118 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:03.382146 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:03.382630 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:03.383050 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:03.383055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:03.383059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:03.383268 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:03.383832 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:03.383841 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:03.384302 (+ 461us) client_negotiation.cc:769] Sending connection context
0619 16:56:03.384594 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 16:56:03.385214 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":559,"threads_started":1}
I20260619 16:56:03.387951 23187 heartbeater.cc:347] Connected to a master server at 127.16.48.62:41377
I20260619 16:56:03.388363 23187 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:03.389153 23187 heartbeater.cc:511] Master 127.16.48.62:41377 requested a full tablet report, sending...
I20260619 16:56:03.390918 22730 ts_manager.cc:194] Registered new tserver with Master: 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621)
I20260619 16:56:03.392096 22730 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:49077
I20260619 16:56:03.392572 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.48.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.61.keytab.
Entry for principal oryx/127.16.48.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.61.keytab.
I20260619 16:56:03.444087 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.61:38415
--webserver_interface=127.16.48.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.61.keytab
--principal=oryx/127.16.48.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:41377,127.16.48.61:38415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:03.796444 23193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:03.796824 23193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:03.796980 23193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:03.804774 23193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:03.804934 23193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:03.805038 23193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:03.805145 23193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:03.805238 23193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:03.815954 23193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:41377,127.16.48.61:38415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.61.keytab
--principal=oryx/127.16.48.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.61:38415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.48.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
W20260619 16:56:03.816442 23193 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.48.62:41377,127.16.48.61:38415'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 16:56:03.817570 23193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:03.819027 23193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:03.832171 23199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:03.832291 23201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:03.833262 23193 server_base.cc:1034] running on GCE node
W20260619 16:56:03.832203 23198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:03.834408 23193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:03.836237 23193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:03.837482 23193 hybrid_clock.cc:648] HybridClock initialized: now 1781888163837416 us; error 44 us; skew 500 ppm
I20260619 16:56:03.837889 23193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:03 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888163, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:03.841419 23193 init.cc:377] Logged in from keytab as oryx/127.16.48.61@KRBTEST.COM (short username oryx)
I20260619 16:56:03.843335 23193 webserver.cc:460] Webserver started at http://127.16.48.61:35995/ using document root <none> and password file <none>
I20260619 16:56:03.844218 23193 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:03.844341 23193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:03.844774 23193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:03.844858 23053 heartbeater.cc:503] Master 127.16.48.62:41377 was elected leader, sending a full tablet report...
I20260619 16:56:03.848462 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data/instance:
uuid: "2c609163a3874286809b0d1adc1405da"
format_stamp: "Formatted at 2026-06-19 16:56:03 on dist-test-slave-3ffd"
server_key: "3243b775c35fe7c95eadb946f1e3955a"
server_key_iv: "570a8c4099cac89d8824a022f740b89d"
server_key_version: "encryptionkey@0"
I20260619 16:56:03.849287 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal/instance:
uuid: "2c609163a3874286809b0d1adc1405da"
format_stamp: "Formatted at 2026-06-19 16:56:03 on dist-test-slave-3ffd"
server_key: "3243b775c35fe7c95eadb946f1e3955a"
server_key_iv: "570a8c4099cac89d8824a022f740b89d"
server_key_version: "encryptionkey@0"
I20260619 16:56:03.855043 23193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:56:03.858986 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:03.860567 23193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:56:03.860785 23193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal
uuid: "2c609163a3874286809b0d1adc1405da"
format_stamp: "Formatted at 2026-06-19 16:56:03 on dist-test-slave-3ffd"
server_key: "3243b775c35fe7c95eadb946f1e3955a"
server_key_iv: "570a8c4099cac89d8824a022f740b89d"
server_key_version: "encryptionkey@0"
I20260619 16:56:03.861008 23193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:03.889247 23193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:03.894459 23193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:03.894779 23193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:03.913894 23193 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.61:38415
I20260619 16:56:03.913914 23259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.61:38415 every 8 connection(s)
I20260619 16:56:03.915395 23193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
I20260619 16:56:03.916724 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 23193
I20260619 16:56:03.920631 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:03.928191 23260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:56:03 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.61@KRBTEST.COM
Jun 19 16:56:03 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888163, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.61@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:03.941531 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:03.919081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.16.48.61:38415)
0619 16:56:03.919974 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:03.919987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:03.920029 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:56:03.921822 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:03.921846 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:03.922262 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:03.922693 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:03.924353 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.925812 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.927177 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.927888 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.928108 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:03.933079 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:03.933087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:03.933102 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:03.933148 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:03.936563 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:03.937371 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:03.937381 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:03.937390 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:03.937664 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:03.938189 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:03.938195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:03.938199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:03.938869 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:03.939134 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:03.939646 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 16:56:03.940111 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":355,"threads_started":1}
I20260619 16:56:03.947563 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:03.930992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.16.48.62:41377)
0619 16:56:03.931505 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:03.931511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:03.932103 (+ 592us) server_negotiation.cc:407] Connection header received
0619 16:56:03.932364 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:03.932373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:03.932478 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:03.932603 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:03.933890 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.935172 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.936921 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.937389 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.937527 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:03.941536 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:03.941544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:03.941552 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:03.941642 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:03.945102 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:03.945904 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:03.945908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:03.945911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:03.946025 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:03.946335 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:03.946338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:03.946340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:03.946660 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:03.946767 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:03.947173 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 16:56:03.947337 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":258,"threads_started":1}
I20260619 16:56:03.947662 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:03.930996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:59800)
0619 16:56:03.931935 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:03.931991 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:03.932212 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:03.932666 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:03.932671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:03.933359 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:03.933704 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:03.933718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.935321 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.935325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:03.936765 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:03.936780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.937608 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.937614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:03.937784 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:03.938545 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:03.938563 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:03.941263 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:03.945288 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:03.945297 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:03.945302 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:03.945784 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:03.946117 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:03.946120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:03.946122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:03.946241 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:03.946776 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:03.946781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:03.946926 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 16:56:03.947077 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 16:56:03.947326 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":526,"threads_started":1}
I20260619 16:56:03.949559 23260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:56:03 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888163, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.61@KRBTEST.COM for oryx/127.16.48.61@KRBTEST.COM
I20260619 16:56:03.963006 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:03.950293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.0.0.1:60814)
0619 16:56:03.950526 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:03.950547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:03.950678 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:03.950934 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:03.950937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:03.951294 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:03.951430 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:03.951438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.952511 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.952515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:03.953381 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:03.953392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.954038 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.954043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:03.954164 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:03.954726 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:03.954743 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:03.956900 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:03.959803 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:03.959807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:03.959810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:03.960285 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:03.960801 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:03.960806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:03.960809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:03.961055 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:03.961768 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:03.961773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:03.961957 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 16:56:03.962128 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 16:56:03.962566 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 16:56:03.963232 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:03.950503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.16.48.61:38415)
0619 16:56:03.950674 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:03.950678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:03.950694 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:56:03.950770 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:03.950774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:03.950825 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:03.950898 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:03.951552 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.952356 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.953504 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:03.953896 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:03.953986 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:03.957088 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:03.957093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:03.957096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:03.957127 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:03.959635 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:03.960392 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:03.960397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:03.960399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:03.960638 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:03.961184 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:03.961193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:03.961198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:03.961599 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:03.961805 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:03.962243 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 16:56:03.962799 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":234,"server-negotiator.queue_time_us":65}
I20260619 16:56:03.968901 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: Bootstrap starting.
I20260619 16:56:03.972230 23260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:03.973335 23260 log.cc:824] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:03.976379 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: No bootstrap required, opened a new log
I20260619 16:56:03.983565 23260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } }
I20260619 16:56:03.983935 23260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:03.984009 23260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c609163a3874286809b0d1adc1405da, State: Initialized, Role: FOLLOWER
I20260619 16:56:03.984660 23260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } }
I20260619 16:56:03.985562 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } } }
I20260619 16:56:03.985944 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:56:03.986601 23260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:56:03.996074 23278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:56:03.996404 23278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:56:03.998983 23193 master_runner.cc:428] Detected that this master 2c609163a3874286809b0d1adc1405da is joining an existing cluster
I20260619 16:56:03.999102 23193 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.48.61:38415
WARNING: no policy specified for oryx/127.16.48.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.60@KRBTEST.COM" created.
I20260619 16:56:04.029142 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.010406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:59812)
0619 16:56:04.012726 (+ 2320us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.012749 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.013007 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.013646 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.013651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.014349 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.014753 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.014766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.016313 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.016321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.018066 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.018083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.019427 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.019435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.019615 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.020363 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.020388 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.021563 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.025927 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.025933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.025937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.026448 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.026966 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.026973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.026981 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.027147 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.027821 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.027828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.028033 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.028214 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.028572 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2139,"thread_start_us":1932,"threads_started":1}
I20260619 16:56:04.030225 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.010455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.16.48.62:41377)
0619 16:56:04.010649 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.010656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.012864 (+ 2208us) server_negotiation.cc:407] Connection header received
0619 16:56:04.013208 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.013215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.013323 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.013416 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.014953 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.016077 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.018265 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.018943 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.019053 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.021832 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.021842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.021849 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.021892 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.025767 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.026644 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.026649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.026653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.026822 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.027287 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.027291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.027294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.027647 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.029051 (+ 1404us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.029511 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.029802 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:56:04.038273 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.013180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.16.48.61:38415)
0619 16:56:04.013357 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.013362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.022078 (+ 8716us) server_negotiation.cc:407] Connection header received
0619 16:56:04.022300 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.022306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.022371 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.022434 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.023840 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.025043 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.027278 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.027992 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.028092 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.030949 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.030956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.030960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.030995 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.035002 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.035930 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.035936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.035942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.036153 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.036587 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.036593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.036596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.036986 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.037108 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.037595 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.037837 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:56:04.039338 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.013466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.0.0.1:60830)
0619 16:56:04.021990 (+ 8524us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.022012 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.022177 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.022630 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.022635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.023283 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.023651 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.023664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.025308 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.025314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.027123 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.027136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.028426 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.028432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.028601 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.029619 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.029641 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.030730 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.035183 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.035189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.035195 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.035783 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.036296 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.036302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.036305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.036460 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.037207 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.037212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.037355 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.038556 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.038881 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8357,"thread_start_us":6637,"threads_started":1}
Jun 19 16:56:04 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.62@KRBTEST.COM for oryx/127.16.48.61@KRBTEST.COM
I20260619 16:56:04.062269 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.037759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.0.0.1:60842)
0619 16:56:04.041985 (+ 4226us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.042045 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.042335 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.043187 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.043196 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.044196 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.044515 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.044530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.046117 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.046122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.047979 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.048007 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.049486 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.049492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.049731 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.050629 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.050655 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.054346 (+ 3691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.058971 (+ 4625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.058992 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.059001 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.059741 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.060309 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.060315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.060322 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.060539 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.061218 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.061225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.061406 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.061608 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.061952 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3900,"thread_start_us":3771,"threads_started":1}
I20260619 16:56:04.062527 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.038177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.16.48.61:38415)
0619 16:56:04.038518 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.038523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.042155 (+ 3632us) server_negotiation.cc:407] Connection header received
0619 16:56:04.042663 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.042669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.042811 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.042887 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.044717 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.045926 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.048253 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.048985 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.049116 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.054636 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.054644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.054649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.054701 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.058742 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.059945 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.059951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.059955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.060172 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.060686 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.060692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.060696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.061064 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.061236 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.061800 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.062056 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":14952,"threads_started":1}
I20260619 16:56:04.064869 22730 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.48.61:38415: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2c609163a3874286809b0d1adc1405da"
member_type: NON_VOTER
last_known_addr {
host: "127.16.48.61"
port: 38415
}
attrs {
promote: true
}
}
dest_uuid: "7789317184b442768fe9397c633f7ee6"
cas_config_opid_index: -1
I20260619 16:56:04.066105 22730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.60.keytab.
Entry for principal oryx/127.16.48.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.60.keytab.
I20260619 16:56:04.069456 23292 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2c609163a3874286809b0d1adc1405da (127.16.48.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } }
I20260619 16:56:04.071636 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.60:37155
--webserver_interface=127.16.48.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.60.keytab
--principal=oryx/127.16.48.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:41377,127.16.48.61:38415,127.16.48.60:37155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:56:04.072958 23235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:04.074184 23292 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.48.61:38415
I20260619 16:56:04.074842 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } }
I20260619 16:56:04.074998 23193 master.cc:537] Master@127.16.48.61:38415 shutting down...
I20260619 16:56:04.075075 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:04.080909 23235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Refusing update from remote peer 7789317184b442768fe9397c633f7ee6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 16:56:04.082228 23292 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:56:04.085426 22716 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 16:56:04.086856 23193 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:04.087321 23193 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:04.087525 23193 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: stopping tablet replica
I20260619 16:56:04.096566 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.096064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.0.0.1:60852)
0619 16:56:04.096266 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.096418 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:56:04.096762 23193 master.cc:552] Master@127.16.48.61:38415 shutdown complete.
I20260619 16:56:04.096854 23193 master_runner.cc:305] Clearing existing system tablet
I20260619 16:56:04.097265 23193 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:04.097393 23193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260619 16:56:04.099718 22716 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 -> Peer 2c609163a3874286809b0d1adc1405da (127.16.48.61:38415): Couldn't send request to peer 2c609163a3874286809b0d1adc1405da. Status: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:56:04.100518 23193 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:56:04.102092 23297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:04.102406 23193 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:04.102723 23193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:56:04.102813 23193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal
uuid: "2c609163a3874286809b0d1adc1405da"
format_stamp: "Formatted at 2026-06-19 16:56:03 on dist-test-slave-3ffd"
server_key: "3243b775c35fe7c95eadb946f1e3955a"
server_key_iv: "570a8c4099cac89d8824a022f740b89d"
server_key_version: "encryptionkey@0"
I20260619 16:56:04.106455 23193 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:04.111963 23193 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:56:04.112202 23193 log.cc:1196] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 16:56:04.112602 23193 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: Deleting consensus metadata
I20260619 16:56:04.112882 23193 master_runner.cc:315] Copying system tablet from 127.16.48.62:41377
I20260619 16:56:04.115502 23193 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.48.62:41377
I20260619 16:56:04.128816 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.116367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.16.48.62:41377)
0619 16:56:04.116746 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.116753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.117069 (+ 316us) server_negotiation.cc:407] Connection header received
0619 16:56:04.117309 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.117315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.117391 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.117498 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.118552 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.119351 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.120762 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.121240 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.121380 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.123289 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.123293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.123297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.123331 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.126069 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.126797 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.126806 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.126810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.126960 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.127342 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.127349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.127353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.127630 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.127787 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.128228 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.128434 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260619 16:56:04.129012 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.116357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:59816)
0619 16:56:04.116934 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.116968 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.117150 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.117530 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.117534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.118016 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.118256 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.118266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.119509 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.119515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.120604 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.120617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.121415 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.121421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.121607 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.122303 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.122322 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.123054 (+ 732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.126253 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.126259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.126263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.126638 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.127083 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.127088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.127092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.127233 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.127754 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.127758 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.127923 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.128070 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.128417 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":387,"threads_started":1}
I20260619 16:56:04.130229 22750 tablet_copy_service.cc:140] P 7789317184b442768fe9397c633f7ee6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2c609163a3874286809b0d1adc1405da ({username='oryx', principal='oryx/127.16.48.61@KRBTEST.COM'} at 127.0.0.1:59816)
I20260619 16:56:04.130515 22750 tablet_copy_service.cc:161] P 7789317184b442768fe9397c633f7ee6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2c609163a3874286809b0d1adc1405da at {username='oryx', principal='oryx/127.16.48.61@KRBTEST.COM'} at 127.0.0.1:59816: session id = 2c609163a3874286809b0d1adc1405da-00000000000000000000000000000000
I20260619 16:56:04.133643 22750 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:56:04.135846 23193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:56:04.140731 23193 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: tablet copy: Starting download of 0 data blocks...
I20260619 16:56:04.141146 23193 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: tablet copy: Starting download of 1 WAL segments...
I20260619 16:56:04.144001 23193 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:56:04.147532 22750 tablet_copy_service.cc:342] P 7789317184b442768fe9397c633f7ee6: Request end of tablet copy session 2c609163a3874286809b0d1adc1405da-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.48.61@KRBTEST.COM'} at 127.0.0.1:59816
I20260619 16:56:04.147761 22750 tablet_copy_service.cc:433] P 7789317184b442768fe9397c633f7ee6: ending tablet copy session 2c609163a3874286809b0d1adc1405da-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2c609163a3874286809b0d1adc1405da
I20260619 16:56:04.160890 23193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:04.163883 23304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:04.170019 23305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:04.177380 23307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:04.177640 23193 server_base.cc:1034] running on GCE node
I20260619 16:56:04.178066 23193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:04.178766 23193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:04.180078 23193 hybrid_clock.cc:648] HybridClock initialized: now 1781888164180010 us; error 51 us; skew 500 ppm
I20260619 16:56:04.180373 23193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:04 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888164, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:04.182426 23193 init.cc:377] Logged in from keytab as oryx/127.16.48.61@KRBTEST.COM (short username oryx)
I20260619 16:56:04.183835 23193 webserver.cc:460] Webserver started at http://127.16.48.61:46839/ using document root <none> and password file <none>
I20260619 16:56:04.184247 23193 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:04.184347 23193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:04.186888 23193 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:56:04.188759 23314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:04.189234 23193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:56:04.189404 23193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal
uuid: "2c609163a3874286809b0d1adc1405da"
format_stamp: "Formatted at 2026-06-19 16:56:03 on dist-test-slave-3ffd"
server_key: "3243b775c35fe7c95eadb946f1e3955a"
server_key_iv: "570a8c4099cac89d8824a022f740b89d"
server_key_version: "encryptionkey@0"
I20260619 16:56:04.189612 23193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:04.204419 23193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:04.207999 23193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:04.250226 23193 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.61:38415
I20260619 16:56:04.250268 23365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.61:38415 every 8 connection(s)
I20260619 16:56:04.251394 23193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
I20260619 16:56:04.258723 23366 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:56:04.269688 23366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: Bootstrap starting.
I20260619 16:56:04.283761 23366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:56:04.284441 23366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: Bootstrap complete.
I20260619 16:56:04.285339 23366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } }
I20260619 16:56:04.285585 23366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c609163a3874286809b0d1adc1405da, State: Initialized, Role: LEARNER
I20260619 16:56:04.285820 23366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } }
I20260619 16:56:04.287109 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } }
I20260619 16:56:04.287451 23366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:56:04.287496 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: This master's current role is: LEARNER
I20260619 16:56:04.297622 23380 catalog_manager.cc:1259] Loaded cluster ID: b3fe076130634d9a832f2c816f6e384e
I20260619 16:56:04.297752 23380 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: loading cluster ID for follower catalog manager: success
I20260619 16:56:04.301854 23380 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: acquiring CA information for follower catalog manager: success
I20260619 16:56:04.305177 23380 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:56:04.395262 23187 heartbeater.cc:503] Master 127.16.48.62:41377 was elected leader, sending a full tablet report...
I20260619 16:56:04.495216 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.485411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.16.48.61:38415)
0619 16:56:04.486339 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.486346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.486372 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:56:04.486461 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.486467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.486566 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.486714 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:04.487316 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.489167 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.492464 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.493816 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.493990 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.494283 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.494679 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.494837 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":541,"threads_started":1}
I20260619 16:56:04.495443 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.485199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.0.0.1:60858)
0619 16:56:04.485524 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.485548 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.485689 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.486809 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.486815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.486852 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:04.487078 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.487091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.489389 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.489396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.492223 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.492250 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.494075 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.494085 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.494425 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.494534 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.494649 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.494814 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260619 16:56:04.497380 23341 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 16:56:04.498412 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7789317184b442768fe9397c633f7ee6. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } }
I20260619 16:56:04.498703 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: This master's current role is: LEARNER
W20260619 16:56:04.524717 23294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:04.525038 23294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:04.525127 23294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:04.533015 23294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:04.533142 23294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:04.533200 23294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:04.533257 23294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:04.533300 23294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:04.544310 23294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:41377,127.16.48.61:38415,127.16.48.60:37155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.60.keytab
--principal=oryx/127.16.48.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.60:37155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.48.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:04.545722 23294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:04.547139 23294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:04.560585 23386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:04.560690 23387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:04.560611 23389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:04.562927 23294 server_base.cc:1034] running on GCE node
I20260619 16:56:04.563836 23294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:04.565547 23294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:04.566764 23294 hybrid_clock.cc:648] HybridClock initialized: now 1781888164566719 us; error 43 us; skew 500 ppm
I20260619 16:56:04.567166 23294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:04 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888164, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:04.570557 23294 init.cc:377] Logged in from keytab as oryx/127.16.48.60@KRBTEST.COM (short username oryx)
I20260619 16:56:04.572347 23294 webserver.cc:460] Webserver started at http://127.16.48.60:35397/ using document root <none> and password file <none>
I20260619 16:56:04.573098 23294 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:04.573212 23294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:04.573478 23294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:04.576797 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data/instance:
uuid: "693be2ed13624107816f36c365395757"
format_stamp: "Formatted at 2026-06-19 16:56:04 on dist-test-slave-3ffd"
server_key: "42ea98829983cf27b23128e1b18dced4"
server_key_iv: "ce0c415466744fcececd5c3ace6ebd38"
server_key_version: "encryptionkey@0"
I20260619 16:56:04.577611 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal/instance:
uuid: "693be2ed13624107816f36c365395757"
format_stamp: "Formatted at 2026-06-19 16:56:04 on dist-test-slave-3ffd"
server_key: "42ea98829983cf27b23128e1b18dced4"
server_key_iv: "ce0c415466744fcececd5c3ace6ebd38"
server_key_version: "encryptionkey@0"
I20260619 16:56:04.583401 23294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:56:04.587496 23396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:04.589385 23294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260619 16:56:04.589615 23294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal
uuid: "693be2ed13624107816f36c365395757"
format_stamp: "Formatted at 2026-06-19 16:56:04 on dist-test-slave-3ffd"
server_key: "42ea98829983cf27b23128e1b18dced4"
server_key_iv: "ce0c415466744fcececd5c3ace6ebd38"
server_key_version: "encryptionkey@0"
I20260619 16:56:04.589811 23294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:04.608362 23294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:04.612808 23294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:04.613133 23294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:04.634429 23294 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.60:37155
I20260619 16:56:04.634488 23447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.60:37155 every 8 connection(s)
I20260619 16:56:04.636196 23294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
I20260619 16:56:04.639005 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 23294
I20260619 16:56:04.640426 23448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:04.650966 23448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:56:04 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.60@KRBTEST.COM
Jun 19 16:56:04 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888164, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.60@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:04.667734 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.641151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.16.48.60:37155)
0619 16:56:04.642569 (+ 1418us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.642587 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.644707 (+ 2120us) server_negotiation.cc:407] Connection header received
0619 16:56:04.646488 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.646516 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.647023 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.647527 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.649126 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.650796 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.652382 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.653137 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.653350 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.658764 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.658775 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.658792 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.658840 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.662470 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.663424 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.663431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.663438 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.663617 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.664071 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.664077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.664082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.664947 (+ 865us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.665283 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.665695 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.666213 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":805,"threads_started":1}
I20260619 16:56:04.670516 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.654003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.16.48.62:41377)
0619 16:56:04.654607 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.654615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.655139 (+ 524us) server_negotiation.cc:407] Connection header received
0619 16:56:04.655359 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.655364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.655441 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.655562 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.656767 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.657873 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.659744 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.660255 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.660403 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.663826 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.663831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.663834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.663868 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.667460 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.668377 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.668384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.668389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.668532 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.668939 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.668947 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.668955 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.669225 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.669389 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.669869 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.670104 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":311,"threads_started":1}
I20260619 16:56:04.670549 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.653919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:59822)
0619 16:56:04.654993 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.655030 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.655232 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.655632 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.655640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.656361 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.656613 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.656622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.658054 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.658058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.659597 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.659609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.660459 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.660465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.660652 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.661194 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.661210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.663612 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.667614 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.667624 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.667630 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.668238 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.668662 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.668667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.668671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.668802 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.669363 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.669370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.669552 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.669767 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.670109 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":715,"threads_started":1}
I20260619 16:56:04.672408 23448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:56:04 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888164, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.60@KRBTEST.COM for oryx/127.16.48.61@KRBTEST.COM
I20260619 16:56:04.685454 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.673026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.0.0.1:60874)
0619 16:56:04.673328 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.673343 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.673427 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.673715 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.673718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.674043 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.674195 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.674202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.675185 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.675189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.676180 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.676212 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.676913 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.676917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.677061 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.677749 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.677768 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.679888 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.683033 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.683037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.683040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.683453 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.683903 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.683907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.683909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.684053 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.684707 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.684711 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.684841 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.684961 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.685215 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260619 16:56:04.685565 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.673209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.16.48.61:38415)
0619 16:56:04.673423 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.673427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.673445 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:56:04.673530 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.673533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.673593 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.673680 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.674325 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.675067 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.676346 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.676772 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.676883 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.680227 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.680234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.680237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.680276 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.682856 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.683604 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.683609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.683611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.683786 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.684244 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.684249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.684251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.684582 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.684705 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.685089 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.685306 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 16:56:04.686522 23448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:56:04 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888164, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.60@KRBTEST.COM for oryx/127.16.48.60@KRBTEST.COM
I20260619 16:56:04.702957 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.687253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:37155 (local address 127.0.0.1:48064)
0619 16:56:04.687573 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.687594 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.687704 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.688029 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.688033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.688572 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.688722 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.688730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.689717 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.689725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.690739 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.690757 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.691463 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.691468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.691677 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.692297 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.692314 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.694425 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.699566 (+ 5141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.699571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.699577 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.700559 (+ 982us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.701018 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.701025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.701029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.701239 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.701951 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.701955 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.702112 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.702264 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.702552 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 16:56:04.703903 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.687509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.16.48.60:37155)
0619 16:56:04.687665 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.687670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.687686 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:56:04.687827 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.687832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.687894 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.688001 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.688851 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.689599 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.690886 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.691292 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.691438 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.694651 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.694656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.694659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.694687 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.699349 (+ 4662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.700715 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.700724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.700727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.700889 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.701357 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.701361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.701364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.701760 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.701903 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.703371 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.703609 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:56:04.712405 23448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Bootstrap starting.
I20260619 16:56:04.715867 23448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:04.717108 23448 log.cc:824] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:04.719965 23448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: No bootstrap required, opened a new log
I20260619 16:56:04.728012 23448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } }
I20260619 16:56:04.728482 23448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:04.728644 23448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693be2ed13624107816f36c365395757, State: Initialized, Role: FOLLOWER
I20260619 16:56:04.729290 23448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } }
I20260619 16:56:04.730206 23455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } } }
I20260619 16:56:04.730619 23455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:56:04.731336 23448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:56:04.741561 23466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:56:04.741770 23466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:56:04.746551 23294 master_runner.cc:428] Detected that this master 693be2ed13624107816f36c365395757 is joining an existing cluster
I20260619 16:56:04.746747 23294 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.48.60:37155
I20260619 16:56:04.770654 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.751771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.16.48.62:41377)
0619 16:56:04.751996 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.752005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.752712 (+ 707us) server_negotiation.cc:407] Connection header received
0619 16:56:04.752991 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.752997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.753086 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.753161 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.755072 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.756471 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.758719 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.759377 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.759484 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.762712 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.762720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.762725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.762770 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.766657 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.767689 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.767695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.767699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.767860 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.768827 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.768832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.768836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.769257 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.769388 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.770018 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.770283 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:56:04.771766 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.751767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:59836)
0619 16:56:04.752573 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.752596 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.752833 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.753384 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.753389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.754499 (+ 1110us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.754864 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.754879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.756689 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.756694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.758524 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.758539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.760234 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.760242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.760457 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.761228 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.761254 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.762448 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.766876 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.766881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.766886 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.767521 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.768460 (+ 939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.768464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.768469 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.768675 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.769484 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.769489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.769723 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.770931 (+ 1208us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.771278 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":32,"spinlock_wait_cycles":29440,"thread_start_us":445,"threads_started":1}
I20260619 16:56:04.773866 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 22791
I20260619 16:56:04.778075 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.752876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.16.48.61:38415)
0619 16:56:04.753115 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.753124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.755610 (+ 2486us) server_negotiation.cc:407] Connection header received
0619 16:56:04.755868 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.755876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.755980 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.756070 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.758092 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.759485 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.762618 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.763539 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.763683 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.767237 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.767246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.767253 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.767305 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.771336 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.772242 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.772247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.772250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.774908 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.776132 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.776138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.776158 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.776601 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.776777 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.777410 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.777691 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:56:04.779264 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.752740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.0.0.1:60882)
0619 16:56:04.755500 (+ 2760us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.755520 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.755711 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.756379 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.756387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.757345 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.757870 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.757888 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.759733 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.759742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.762383 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.762405 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.764071 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.764081 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.764292 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.765346 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.765378 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.766963 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.771520 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.771534 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.771538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.772057 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.775799 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.775802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.775805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.775947 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.776863 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.776867 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.777112 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.778321 (+ 1209us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.778726 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2601,"mutex_wait_us":4,"thread_start_us":2410,"threads_started":1}
I20260619 16:56:04.786474 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.755614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.16.48.60:37155)
0619 16:56:04.756303 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.756309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.767973 (+ 11664us) server_negotiation.cc:407] Connection header received
0619 16:56:04.768049 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.768054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.768165 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.768275 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.769607 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.771236 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.774171 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.774904 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.775016 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.778163 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.778172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.778194 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.778241 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.783209 (+ 4968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.784153 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.784160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.784168 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.784333 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.784795 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.784800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.784803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.785219 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.785348 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.785765 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.786093 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":541}
I20260619 16:56:04.786499 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.755772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:37155 (local address 127.0.0.1:48076)
0619 16:56:04.767711 (+ 11939us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.767733 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.767865 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.768333 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.768340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.769034 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.769446 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.769462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.771810 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.771815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.773907 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.773923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.775341 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.775349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.775514 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.776408 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.776442 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.777922 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.783423 (+ 5501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.783431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.783436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.783986 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.784500 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.784505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.784509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.784655 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.785393 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.785396 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.785525 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.785660 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.785971 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11759,"thread_start_us":12195,"threads_started":1}
Jun 19 16:56:04 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.62@KRBTEST.COM for oryx/127.16.48.60@KRBTEST.COM
I20260619 16:56:04.800380 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.777953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:37155 (local address 127.0.0.1:48078)
0619 16:56:04.778320 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.778340 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.778442 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.783141 (+ 4699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.783146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.783560 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.783723 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.783732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.785332 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.785339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.786738 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.786756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.787853 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.787860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.788054 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.789142 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.789186 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.792440 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.797332 (+ 4892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.797338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.797343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.797886 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.798406 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.798413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.798417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.798580 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.799264 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.799272 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.799461 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.799654 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.799977 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260619 16:56:04.800503 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.778790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.16.48.60:37155)
0619 16:56:04.782298 (+ 3508us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.782302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.782745 (+ 443us) server_negotiation.cc:407] Connection header received
0619 16:56:04.782878 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.782882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.782953 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.783099 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.783915 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.785157 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.786900 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.787670 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.787803 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.792666 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.792671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.792680 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.792713 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.797132 (+ 4419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.798084 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.798090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.798094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.798266 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.798724 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.798729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.798732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.799106 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.799243 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.799757 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.799980 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3405,"thread_start_us":3244,"threads_started":1}
I20260619 16:56:04.802122 22730 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.48.60:37155: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "693be2ed13624107816f36c365395757"
member_type: NON_VOTER
last_known_addr {
host: "127.16.48.60"
port: 37155
}
attrs {
promote: true
}
}
dest_uuid: "7789317184b442768fe9397c633f7ee6"
cas_config_opid_index: 5
I20260619 16:56:04.803314 22730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } }
Jun 19 16:56:04 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.1@KRBTEST.COM
I20260619 16:56:04.805626 23292 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 693be2ed13624107816f36c365395757 (127.16.48.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:04.810956 23341 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 LEARNER]: Refusing update from remote peer 7789317184b442768fe9397c633f7ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:56:04.812417 23423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:04.817582 23423 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 FOLLOWER]: Refusing update from remote peer 7789317184b442768fe9397c633f7ee6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:56:04.824276 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.774988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.16.48.1:39311)
0619 16:56:04.792695 (+ 17707us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.792702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.792728 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:56:04.792784 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.792790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.793022 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.793155 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.795705 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.797119 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.798502 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.799115 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.799335 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.805852 (+ 6517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.805878 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.805908 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.805983 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.817660 (+ 11677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.819225 (+ 1565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.819240 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.819248 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.819508 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.820101 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.820109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.820243 (+ 134us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.820849 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.821269 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.823141 (+ 1872us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.823573 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17502,"thread_start_us":17309,"threads_started":1}
I20260619 16:56:04.826946 22914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:04.827431 23479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 16:56:04.829110 23479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:04.840379 23292 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.48.60:37155
I20260619 16:56:04.841509 23294 master.cc:537] Master@127.16.48.60:37155 shutting down...
I20260619 16:56:04.843694 23479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:04.844333 23479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:04.852567 23341 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 693be2ed13624107816f36c365395757 (127.16.48.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:04.860525 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:04.861042 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: This master's current role is: LEARNER
I20260619 16:56:04.862833 23478 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: attempting to promote NON_VOTER 2c609163a3874286809b0d1adc1405da to VOTER
I20260619 16:56:04.863379 23423 raft_consensus.cc:858] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } }
I20260619 16:56:04.863859 23478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } }
I20260619 16:56:04.867918 23341 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 LEARNER]: Refusing update from remote peer 7789317184b442768fe9397c633f7ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 16:56:04.868965 23478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 16:56:04.872094 23479 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2c609163a3874286809b0d1adc1405da (127.16.48.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:04.876411 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:04.876924 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:04.876762 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:04.877115 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:04.877851 23341 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2c609163a3874286809b0d1adc1405da (127.16.48.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
W20260619 16:56:04.863605 23423 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } }
W20260619 16:56:04.882392 23402 tablet.cc:2369] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:56:04.883009 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:04.883315 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:56:04.896183 22716 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 -> Peer 693be2ed13624107816f36c365395757 (127.16.48.60:37155): Couldn't send request to peer 693be2ed13624107816f36c365395757. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:56:04.901515 23423 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2c609163a3874286809b0d1adc1405da (127.16.48.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } }
I20260619 16:56:04.906775 23486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } }
I20260619 16:56:04.906852 23455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7789317184b442768fe9397c633f7ee6. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } }
I20260619 16:56:04.907104 23486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:56:04.907351 23455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:56:04.907631 23489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: NON_VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: true } } }
I20260619 16:56:04.908149 23489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:56:04.908778 23294 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 16:56:04.909229 23294 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 16:56:04.909421 23294 tablet_replica.cc:330] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: stopping tablet replica
I20260619 16:56:04.926928 23294 master.cc:552] Master@127.16.48.60:37155 shutdown complete.
I20260619 16:56:04.927058 23294 master_runner.cc:305] Clearing existing system tablet
I20260619 16:56:04.927417 23294 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:04.927534 23294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:04.930209 23294 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260619 16:56:04.931949 23491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:04.932392 23294 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:04.932710 23294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 16:56:04.932813 23294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal
uuid: "693be2ed13624107816f36c365395757"
format_stamp: "Formatted at 2026-06-19 16:56:04 on dist-test-slave-3ffd"
server_key: "42ea98829983cf27b23128e1b18dced4"
server_key_iv: "ce0c415466744fcececd5c3ace6ebd38"
server_key_version: "encryptionkey@0"
I20260619 16:56:04.936792 23294 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:04.941988 23294 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:56:04.942217 23294 log.cc:1196] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 16:56:04.943091 23294 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Deleting consensus metadata
I20260619 16:56:04.943399 23294 master_runner.cc:315] Copying system tablet from 127.16.48.62:41377
I20260619 16:56:04.948190 23294 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: tablet copy: Beginning tablet copy session from remote peer at address 127.16.48.62:41377
I20260619 16:56:04.969175 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.949100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.16.48.62:41377)
0619 16:56:04.949371 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:04.949380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:04.956260 (+ 6880us) server_negotiation.cc:407] Connection header received
0619 16:56:04.956412 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:04.956416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:04.956508 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:04.956625 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:04.957529 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.958285 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.959612 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.960029 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.960196 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.962292 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:04.962297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:04.962301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:04.962334 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:04.965621 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.966590 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.966596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.966599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.966778 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:04.967283 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:04.967288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:04.967294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:04.967675 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:04.967825 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:04.968525 (+ 700us) server_negotiation.cc:299] Negotiation successful
0619 16:56:04.968773 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 16:56:04.969195 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:04.949061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:59846)
0619 16:56:04.956014 (+ 6953us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:04.956034 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:04.956260 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:04.956663 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:04.956668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:04.957122 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:04.957340 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.957353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.958443 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.958450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.959448 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:04.959461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:04.960222 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:04.960228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:04.960433 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:04.961142 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:04.961159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:04.962070 (+ 911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:04.965829 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.965838 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.965841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.966411 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.966948 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:04.966953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:04.966957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:04.967136 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:04.967971 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:04.967974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:04.968249 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 16:56:04.968417 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 16:56:04.968776 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6784,"thread_start_us":6630,"threads_started":1}
I20260619 16:56:04.970187 22750 tablet_copy_service.cc:140] P 7789317184b442768fe9397c633f7ee6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 693be2ed13624107816f36c365395757 ({username='oryx', principal='oryx/127.16.48.60@KRBTEST.COM'} at 127.0.0.1:59846)
I20260619 16:56:04.970374 22750 tablet_copy_service.cc:161] P 7789317184b442768fe9397c633f7ee6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 693be2ed13624107816f36c365395757 at {username='oryx', principal='oryx/127.16.48.60@KRBTEST.COM'} at 127.0.0.1:59846: session id = 693be2ed13624107816f36c365395757-00000000000000000000000000000000
I20260619 16:56:04.973470 22750 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:56:04.975283 23294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:56:04.980361 23294 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: tablet copy: Starting download of 0 data blocks...
I20260619 16:56:04.980715 23294 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: tablet copy: Starting download of 1 WAL segments...
I20260619 16:56:04.983153 23294 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:56:04.986786 22750 tablet_copy_service.cc:342] P 7789317184b442768fe9397c633f7ee6: Request end of tablet copy session 693be2ed13624107816f36c365395757-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.48.60@KRBTEST.COM'} at 127.0.0.1:59846
I20260619 16:56:04.986953 22750 tablet_copy_service.cc:433] P 7789317184b442768fe9397c633f7ee6: ending tablet copy session 693be2ed13624107816f36c365395757-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 693be2ed13624107816f36c365395757
I20260619 16:56:04.998379 23294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:05.001250 23498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:05.006068 23499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:05.011592 23501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:05.011865 23294 server_base.cc:1034] running on GCE node
I20260619 16:56:05.012296 23294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:05.012996 23294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:05.014204 23294 hybrid_clock.cc:648] HybridClock initialized: now 1781888165014117 us; error 81 us; skew 500 ppm
I20260619 16:56:05.014461 23294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:05 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888165, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:05.017681 23294 init.cc:377] Logged in from keytab as oryx/127.16.48.60@KRBTEST.COM (short username oryx)
I20260619 16:56:05.017817 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 22791
I20260619 16:56:05.025641 23294 webserver.cc:460] Webserver started at http://127.16.48.60:35961/ using document root <none> and password file <none>
I20260619 16:56:05.026036 23294 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:05.026156 23294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:05.029681 23294 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 16:56:05.031801 23508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:05.032285 23294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:56:05.032414 23294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal
uuid: "693be2ed13624107816f36c365395757"
format_stamp: "Formatted at 2026-06-19 16:56:04 on dist-test-slave-3ffd"
server_key: "42ea98829983cf27b23128e1b18dced4"
server_key_iv: "ce0c415466744fcececd5c3ace6ebd38"
server_key_version: "encryptionkey@0"
I20260619 16:56:05.032605 23294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:05.038105 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:39311
--local_ip_for_outbound_sockets=127.16.48.1
--tserver_master_addrs=127.16.48.62:41377
--webserver_port=35799
--webserver_interface=127.16.48.1
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab
--principal=oryx/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.48.62:41377,127.16.48.61:38415
--tserver_master_addrs=127.16.48.62:41377,127.16.48.61:38415,127.16.48.60:37155 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:56:05.055269 23294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:05.059724 23294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:05.088163 23560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.60:37155 every 8 connection(s)
I20260619 16:56:05.088163 23294 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.60:37155
I20260619 16:56:05.089375 23294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
I20260619 16:56:05.095227 23561 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:56:05.099941 23561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Bootstrap starting.
I20260619 16:56:05.116441 23561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:56:05.117003 23561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Bootstrap complete.
I20260619 16:56:05.118105 23561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } }
I20260619 16:56:05.118415 23561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 693be2ed13624107816f36c365395757, State: Initialized, Role: LEARNER
I20260619 16:56:05.118690 23561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } }
I20260619 16:56:05.119652 23564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:05.119961 23561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:56:05.120186 23564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:56:05.128621 23575 catalog_manager.cc:1259] Loaded cluster ID: b3fe076130634d9a832f2c816f6e384e
I20260619 16:56:05.128731 23575 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: loading cluster ID for follower catalog manager: success
I20260619 16:56:05.132367 23575 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: acquiring CA information for follower catalog manager: success
I20260619 16:56:05.135078 23575 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:56:05.354512 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:05.346777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:37155 (local address 127.0.0.1:48086)
0619 16:56:05.347353 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:05.347384 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:05.347545 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:05.348086 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:05.348090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:05.348170 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:05.348388 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:05.348398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.350406 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.350410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:05.351879 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:05.351897 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.353824 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.353830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:05.353997 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:05.354027 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:56:05.354108 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:56:05.354222 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20260619 16:56:05.354727 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:05.346920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.16.48.60:37155)
0619 16:56:05.347670 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:05.347687 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:05.347711 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:56:05.347797 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:05.347804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:05.347911 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:05.348067 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:05.348545 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.350219 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.352099 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.353573 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.353780 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:05.354071 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:05.354152 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 16:56:05.354316 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":395,"threads_started":1}
I20260619 16:56:05.356019 23536 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 16:56:05.356662 23564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7789317184b442768fe9397c633f7ee6. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:05.356937 23564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: This master's current role is: LEARNER
W20260619 16:56:05.437429 23513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:05.437757 23513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:05.437855 23513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:05.446154 23513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:05.446295 23513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:05.446412 23513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:56:05.456701 23513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab
--principal=oryx/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:39311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=35799
--tserver_master_addrs=127.16.48.62:41377,127.16.48.61:38415,127.16.48.60:37155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:05.458043 23513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:05.459363 23513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:05.474674 23582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:05.474721 23585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:05.474623 23583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:05.475909 23513 server_base.cc:1034] running on GCE node
I20260619 16:56:05.476630 23513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:05.477607 23513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:05.478833 23513 hybrid_clock.cc:648] HybridClock initialized: now 1781888165478780 us; error 42 us; skew 500 ppm
I20260619 16:56:05.479202 23513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:05 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888165, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:05.482690 23513 init.cc:377] Logged in from keytab as oryx/127.16.48.1@KRBTEST.COM (short username oryx)
I20260619 16:56:05.484338 23513 webserver.cc:460] Webserver started at http://127.16.48.1:35799/ using document root <none> and password file <none>
I20260619 16:56:05.485057 23513 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:05.485198 23513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:05.492202 23513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:56:05.495221 23592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:05.496888 23513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:56:05.497089 23513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "7552fdd23713435ab989a0150f76e121"
format_stamp: "Formatted at 2026-06-19 16:56:02 on dist-test-slave-3ffd"
server_key: "ef37236acc2434bac03e913ae76f32b1"
server_key_iv: "ca3b05677aa9f9805d9fa57202197880"
server_key_version: "encryptionkey@0"
I20260619 16:56:05.498191 23513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:05.523860 23513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:05.528860 23513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:05.529214 23513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:05.530362 23513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:05.531831 23513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:05.531919 23513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:05.532066 23513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:05.532169 23513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:05.569599 23513 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:39311
I20260619 16:56:05.569715 23704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:39311 every 8 connection(s)
I20260619 16:56:05.571370 23513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:56:05.573550 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 23513
I20260619 16:56:05.574007 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 22925
Jun 19 16:56:05 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.2@KRBTEST.COM
Jun 19 16:56:05 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888165, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.60@KRBTEST.COM
Jun 19 16:56:05 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888165, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
Jun 19 16:56:05 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888165, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.61@KRBTEST.COM
I20260619 16:56:05.626710 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:05.575175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.16.48.2:40595)
0619 16:56:05.586787 (+ 11612us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:05.586800 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:05.588671 (+ 1871us) server_negotiation.cc:407] Connection header received
0619 16:56:05.589468 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:05.589479 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:05.589837 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:05.590025 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:05.593979 (+ 3954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.597922 (+ 3943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.600730 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.601986 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.602173 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:05.614934 (+ 12761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:05.614953 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:05.614973 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:05.615017 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:05.621139 (+ 6122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:05.622273 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:05.622285 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:05.622289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:05.622465 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:05.623031 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:05.623038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:05.623042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:05.623509 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:05.623815 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:05.625891 (+ 2076us) server_negotiation.cc:299] Negotiation successful
0619 16:56:05.626197 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11409,"thread_start_us":11217,"threads_started":1}
I20260619 16:56:05.627211 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:05.589705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:41407 (local address 127.16.48.60:37155)
0619 16:56:05.589956 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:05.589965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:05.592590 (+ 2625us) server_negotiation.cc:407] Connection header received
0619 16:56:05.604578 (+ 11988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:05.604585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:05.604676 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:05.604744 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:05.607976 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.609132 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.611947 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.612658 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.612756 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:05.619008 (+ 6252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:05.619018 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:05.619025 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:05.619068 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:05.622737 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:05.623797 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:05.623803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:05.623806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:05.623983 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:05.624601 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:05.624606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:05.624610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:05.625406 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:05.625528 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:05.626615 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0619 16:56:05.626863 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:56:05.627583 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:05.574387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:58897 (local address 127.16.48.62:41377)
0619 16:56:05.597667 (+ 23280us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:05.597676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:05.597706 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:56:05.604529 (+ 6823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:05.604536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:05.604649 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:05.604776 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:05.609961 (+ 5185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.611205 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.614112 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.614816 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.614934 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:05.620010 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:05.620020 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:05.620028 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:05.620072 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:05.623672 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:05.624596 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:05.624603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:05.624607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:05.625308 (+ 701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:05.625850 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:05.625856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:05.625859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:05.626219 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:05.626357 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:05.626950 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 16:56:05.627180 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23110,"thread_start_us":23324,"threads_started":1}
I20260619 16:56:05.628556 23048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:05.630177 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:05.575322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.1:58897)
0619 16:56:05.585106 (+ 9784us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:05.585202 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:05.604157 (+ 18955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:05.608702 (+ 4545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:05.608710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:05.609306 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:05.609721 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:05.609738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.611469 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.611476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:05.613922 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:05.613940 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.615235 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.615241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:05.615461 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:05.616636 (+ 1175us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:05.616669 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:05.619744 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:05.623837 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:05.623842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:05.623845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:05.624448 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:05.625489 (+ 1041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:05.625495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:05.625500 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:05.625701 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:05.626441 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:05.626449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:05.626622 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 16:56:05.627811 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0619 16:56:05.628467 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9282,"thread_start_us":12641,"threads_started":1}
I20260619 16:56:05.630234 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:05.589965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:37155 (local address 127.16.48.1:41407)
0619 16:56:05.592228 (+ 2263us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:05.592255 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:05.604172 (+ 11917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:05.605299 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:05.605312 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:05.606743 (+ 1431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:05.607708 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:05.607737 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.609351 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.609356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:05.611743 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:05.611759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.613130 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.613139 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:05.613590 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:05.614475 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:05.614515 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:05.618673 (+ 4158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:05.622917 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:05.622930 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:05.622954 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:05.623652 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:05.624179 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:05.624184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:05.624187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:05.624459 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:05.625654 (+ 1195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:05.625666 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:05.626152 (+ 486us) client_negotiation.cc:769] Sending connection context
0619 16:56:05.628169 (+ 2017us) client_negotiation.cc:240] Negotiation successful
0619 16:56:05.628668 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2078,"mutex_wait_us":180,"thread_start_us":2430,"threads_started":1}
I20260619 16:56:05.631656 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:05.596523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.16.48.1:33021)
0619 16:56:05.610604 (+ 14081us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:05.610639 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:05.610760 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:05.612613 (+ 1853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:05.612618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:05.613269 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:05.613763 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:05.613777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.615223 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.615230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:05.617904 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:05.617932 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.618909 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.618919 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:05.619109 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:05.619769 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:05.619796 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:05.622971 (+ 3175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:05.627797 (+ 4826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:05.627801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:05.627804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:05.628698 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:05.629268 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:05.629273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:05.629279 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:05.629451 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:05.630574 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:05.630581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:05.630772 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 16:56:05.630914 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:56:05.631236 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13925,"mutex_wait_us":63,"thread_start_us":14288,"threads_started":1}
I20260619 16:56:05.632246 23705 heartbeater.cc:347] Connected to a master server at 127.16.48.60:37155
I20260619 16:56:05.632442 23706 heartbeater.cc:347] Connected to a master server at 127.16.48.62:41377
I20260619 16:56:05.632611 23706 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:05.632614 23705 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:05.633703 23705 heartbeater.cc:511] Master 127.16.48.60:37155 requested a full tablet report, sending...
I20260619 16:56:05.635040 23706 heartbeater.cc:511] Master 127.16.48.62:41377 requested a full tablet report, sending...
I20260619 16:56:05.635771 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:05.593678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:33021 (local address 127.16.48.61:38415)
0619 16:56:05.611936 (+ 18258us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:05.611942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:05.611965 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:56:05.612023 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:05.612028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:05.612164 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:05.612272 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:05.613954 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.615037 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.618142 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:05.618670 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:05.618826 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:05.623267 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:05.623273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:05.623278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:05.623325 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:05.627546 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:05.628855 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:05.628861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:05.628864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:05.629056 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:05.629580 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:05.629584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:05.629587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:05.630361 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:05.630513 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:05.631932 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0619 16:56:05.635371 (+ 3439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18120,"thread_start_us":18762,"threads_started":1}
I20260619 16:56:05.636584 23526 ts_manager.cc:194] Registered new tserver with Master: 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311)
I20260619 16:56:05.637148 23707 heartbeater.cc:347] Connected to a master server at 127.16.48.61:38415
I20260619 16:56:05.637311 23707 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:05.637780 23707 heartbeater.cc:511] Master 127.16.48.61:38415 requested a full tablet report, sending...
I20260619 16:56:05.638720 22730 ts_manager.cc:194] Re-registered known tserver with Master: 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311)
I20260619 16:56:05.639982 22730 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:58897
I20260619 16:56:05.642066 23331 ts_manager.cc:194] Registered new tserver with Master: 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311)
I20260619 16:56:05.722488 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 22925
I20260619 16:56:05.726353 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: NON_VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: true } } }
I20260619 16:56:05.726753 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:05.729074 23478 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: attempting to promote NON_VOTER 693be2ed13624107816f36c365395757 to VOTER
I20260619 16:56:05.729980 23478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } }
I20260619 16:56:05.732221 23536 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 LEARNER]: Refusing update from remote peer 7789317184b442768fe9397c633f7ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 16:56:05.732267 23341 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Refusing update from remote peer 7789317184b442768fe9397c633f7ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 16:56:05.732872 23718 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:56:05.733374 23717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:56:05.736629 23478 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 693be2ed13624107816f36c365395757 (127.16.48.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:05.741257 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:05.741571 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:05.741426 23341 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 693be2ed13624107816f36c365395757 (127.16.48.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:05.741868 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:05.742117 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:05.741938 23536 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 693be2ed13624107816f36c365395757 (127.16.48.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:05.742350 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:05.742568 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:05.744694 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:40595
--local_ip_for_outbound_sockets=127.16.48.2
--tserver_master_addrs=127.16.48.62:41377
--webserver_port=39311
--webserver_interface=127.16.48.2
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab
--principal=oryx/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.48.62:41377,127.16.48.61:38415
--tserver_master_addrs=127.16.48.62:41377,127.16.48.61:38415,127.16.48.60:37155 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:56:05.746718 23721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:05.747110 23721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:56:05.747493 23564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7789317184b442768fe9397c633f7ee6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:05.747735 23564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:56:06.093729 23722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:06.094084 23722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:06.094260 23722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:06.102283 23722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:06.102466 23722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:06.102703 23722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:56:06.113276 23722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab
--principal=oryx/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:40595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=39311
--tserver_master_addrs=127.16.48.62:41377,127.16.48.61:38415,127.16.48.60:37155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:06.114758 23722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:06.116407 23722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:06.131314 23730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:06.131356 23728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:06.131757 23727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:06.132863 23722 server_base.cc:1034] running on GCE node
I20260619 16:56:06.133639 23722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:06.134711 23722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:06.135941 23722 hybrid_clock.cc:648] HybridClock initialized: now 1781888166135896 us; error 40 us; skew 500 ppm
I20260619 16:56:06.136396 23722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:06 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888166, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:06.139863 23722 init.cc:377] Logged in from keytab as oryx/127.16.48.2@KRBTEST.COM (short username oryx)
I20260619 16:56:06.141584 23722 webserver.cc:460] Webserver started at http://127.16.48.2:39311/ using document root <none> and password file <none>
I20260619 16:56:06.142418 23722 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:06.142573 23722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:06.148892 23722 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 16:56:06.152061 23737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:06.153636 23722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:56:06.153858 23722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "731e6bc712914e86a3c5307622e64f51"
format_stamp: "Formatted at 2026-06-19 16:56:02 on dist-test-slave-3ffd"
server_key: "69313e8da11e71c4befb8844f6c13a62"
server_key_iv: "adbcb75c2b8e7c187f086d246a7cf2a8"
server_key_version: "encryptionkey@0"
I20260619 16:56:06.154922 23722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:06.174634 23722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:06.179473 23722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:06.179795 23722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:06.180902 23722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:06.182443 23722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:06.182574 23722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:06.182711 23722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:06.182801 23722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:06.218092 23722 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:40595
I20260619 16:56:06.218138 23849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:40595 every 8 connection(s)
I20260619 16:56:06.219666 23722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:56:06.228327 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 23722
I20260619 16:56:06.228621 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 23059
Jun 19 16:56:06 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888166, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.61@KRBTEST.COM
Jun 19 16:56:06 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888166, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
Jun 19 16:56:06 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888166, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.60@KRBTEST.COM
Jun 19 16:56:06 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.3@KRBTEST.COM
I20260619 16:56:06.259537 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.225817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:58423 (local address 127.16.48.60:37155)
0619 16:56:06.226326 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.226334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.226906 (+ 572us) server_negotiation.cc:407] Connection header received
0619 16:56:06.229176 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.229184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.229263 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.229366 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.232708 (+ 3342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.234347 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.240524 (+ 6177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.241192 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.241300 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.249096 (+ 7796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.249103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.249107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.249148 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.253335 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.255283 (+ 1948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.255291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.255297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.255539 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.257664 (+ 2125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.257672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.257678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.258085 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.258226 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.258799 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.259058 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260619 16:56:06.259722 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.224954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:39423)
0619 16:56:06.225621 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:06.225642 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:06.227639 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:06.238019 (+ 10380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:06.238027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:06.238853 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:06.239214 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.239227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.240655 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.240660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.242378 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.242396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.243913 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.243920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.244153 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.244801 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:06.244825 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:06.247495 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:06.255575 (+ 8080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.255580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.255584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.256082 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.256531 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.256536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.256540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.256695 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.257322 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:06.257327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:06.257463 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 16:56:06.257631 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 16:56:06.258221 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":339,"threads_started":1}
I20260619 16:56:06.260944 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.226275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:37155 (local address 127.16.48.2:58423)
0619 16:56:06.226801 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:06.226818 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:06.227639 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:06.229584 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:06.229589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:06.232024 (+ 2435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:06.232470 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.232487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.234712 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.234721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.240370 (+ 5649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.240382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.241628 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.241636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.241829 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.242863 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:06.242900 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:06.248857 (+ 5957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:06.253526 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.253534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.253539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.254212 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.255727 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.255735 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.255738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.255917 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.258308 (+ 2391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:06.258315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:06.258466 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 16:56:06.260231 (+ 1765us) client_negotiation.cc:240] Negotiation successful
0619 16:56:06.260536 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":562,"thread_start_us":287,"threads_started":1}
I20260619 16:56:06.260957 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.225087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:39423 (local address 127.16.48.62:41377)
0619 16:56:06.237709 (+ 12622us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.237715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.237734 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:56:06.237792 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.237796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.237875 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.237986 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.239416 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.240490 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.242636 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.243658 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.244490 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.247803 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.247810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.247815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.247857 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.255336 (+ 7479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.256270 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.256276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.256280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.256419 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.256838 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.256843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.256846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.257185 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.260216 (+ 3031us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.260354 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.260593 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12469,"thread_start_us":12571,"threads_started":1}
I20260619 16:56:06.262147 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.222893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:34925 (local address 127.16.48.61:38415)
0619 16:56:06.223451 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.223455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.224509 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 16:56:06.228054 (+ 3545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.228063 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.228228 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.228333 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.235794 (+ 7461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.237366 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.240306 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.240950 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.241079 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.248145 (+ 7066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.248154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.248160 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.248218 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.253051 (+ 4833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.254486 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.254495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.254500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.254683 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.256076 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.256082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.256086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.256500 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.256622 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.261541 (+ 4919us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.261800 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":328,"threads_started":1}
I20260619 16:56:06.262670 23850 heartbeater.cc:347] Connected to a master server at 127.16.48.60:37155
I20260619 16:56:06.263026 23850 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:06.263135 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.223524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.16.48.2:34925)
0619 16:56:06.224336 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:06.224386 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:06.227625 (+ 3239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:06.229036 (+ 1411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:06.229049 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:06.230205 (+ 1156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:06.231319 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.231354 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.237658 (+ 6304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.237666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.240149 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.240179 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.241434 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.241442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.241834 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.243810 (+ 1976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:06.243842 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:06.247869 (+ 4027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:06.253239 (+ 5370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.253250 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.253271 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.254348 (+ 1077us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.254844 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.254848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.254852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.255137 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.256727 (+ 1590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:06.256737 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:06.257255 (+ 518us) client_negotiation.cc:769] Sending connection context
0619 16:56:06.262367 (+ 5112us) client_negotiation.cc:240] Negotiation successful
0619 16:56:06.262707 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":57,"thread_start_us":299,"threads_started":1}
I20260619 16:56:06.263453 23851 heartbeater.cc:347] Connected to a master server at 127.16.48.62:41377
I20260619 16:56:06.263572 23851 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:06.264106 23851 heartbeater.cc:511] Master 127.16.48.62:41377 requested a full tablet report, sending...
I20260619 16:56:06.264178 23850 heartbeater.cc:511] Master 127.16.48.60:37155 requested a full tablet report, sending...
I20260619 16:56:06.265825 23852 heartbeater.cc:347] Connected to a master server at 127.16.48.61:38415
I20260619 16:56:06.265959 23852 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:06.266067 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.229645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.16.48.3:33621)
0619 16:56:06.240313 (+ 10668us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.240322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.240353 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:56:06.240417 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.240426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.240735 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.243218 (+ 2483us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.243476 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.245089 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.247324 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.248941 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.252716 (+ 3775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.256624 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.256644 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.256665 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.256710 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.261495 (+ 4785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.262494 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.262506 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.262509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.262710 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.263174 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.263178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.263181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.263622 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.263911 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.265324 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.265638 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10450,"thread_start_us":10287,"threads_started":1}
I20260619 16:56:06.266243 22730 ts_manager.cc:194] Re-registered known tserver with Master: 731e6bc712914e86a3c5307622e64f51 (127.16.48.2:40595)
I20260619 16:56:06.266369 23852 heartbeater.cc:511] Master 127.16.48.61:38415 requested a full tablet report, sending...
I20260619 16:56:06.266352 23526 ts_manager.cc:194] Registered new tserver with Master: 731e6bc712914e86a3c5307622e64f51 (127.16.48.2:40595)
I20260619 16:56:06.267414 22730 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:39423
I20260619 16:56:06.267482 23182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:06.267786 23331 ts_manager.cc:194] Registered new tserver with Master: 731e6bc712914e86a3c5307622e64f51 (127.16.48.2:40595)
I20260619 16:56:06.355541 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 23059
I20260619 16:56:06.372566 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:33621
--local_ip_for_outbound_sockets=127.16.48.3
--tserver_master_addrs=127.16.48.62:41377
--webserver_port=37847
--webserver_interface=127.16.48.3
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab
--principal=oryx/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.48.62:41377,127.16.48.61:38415
--tserver_master_addrs=127.16.48.62:41377,127.16.48.61:38415,127.16.48.60:37155 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:06.577858 23701 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.16.48.1:39311
I20260619 16:56:06.643587 23706 heartbeater.cc:503] Master 127.16.48.62:41377 was elected leader, sending a full tablet report...
W20260619 16:56:06.738142 23863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:06.738570 23863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:06.738763 23863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:06.747241 23863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:06.747406 23863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:06.747640 23863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:56:06.758040 23863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab
--principal=oryx/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:33621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=37847
--tserver_master_addrs=127.16.48.62:41377,127.16.48.61:38415,127.16.48.60:37155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:06.759536 23863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:06.761255 23863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:06.776180 23871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:06.776158 23869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:06.776093 23868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:06.776918 23863 server_base.cc:1034] running on GCE node
I20260619 16:56:06.777700 23863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:06.778743 23863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:06.779985 23863 hybrid_clock.cc:648] HybridClock initialized: now 1781888166779930 us; error 57 us; skew 500 ppm
I20260619 16:56:06.780400 23863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:06 dist-test-slave-3ffd krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888166, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:06.783726 23863 init.cc:377] Logged in from keytab as oryx/127.16.48.3@KRBTEST.COM (short username oryx)
I20260619 16:56:06.785459 23863 webserver.cc:460] Webserver started at http://127.16.48.3:37847/ using document root <none> and password file <none>
I20260619 16:56:06.786237 23863 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:06.786410 23863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:06.792676 23863 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:56:06.795822 23878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:06.797514 23863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:56:06.797729 23863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "2d29389ddcd3472c9b6ea7e1cba2078e"
format_stamp: "Formatted at 2026-06-19 16:56:03 on dist-test-slave-3ffd"
server_key: "829a70a80e90dc3aedb715cf59602005"
server_key_iv: "d2cadca786efa38fdaf00b857e37655c"
server_key_version: "encryptionkey@0"
I20260619 16:56:06.798806 23863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:06.827248 23863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:06.832188 23863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:06.832598 23863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:06.833796 23863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:06.835373 23863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:06.835491 23863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:06.835625 23863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:06.835724 23863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:06.870045 23863 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:33621
I20260619 16:56:06.870190 23990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:33621 every 8 connection(s)
I20260619 16:56:06.871646 23863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:56:06.877162 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 23863
Jun 19 16:56:06 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888166, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
Jun 19 16:56:06 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888166, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.61@KRBTEST.COM
I20260619 16:56:06.907145 23995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.874800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:34353 (local address 127.16.48.62:41377)
0619 16:56:06.875240 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.875244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.876496 (+ 1252us) server_negotiation.cc:407] Connection header received
0619 16:56:06.879877 (+ 3381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.879887 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.880013 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.880142 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.883322 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.884906 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.888327 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.888978 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.889079 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.895195 (+ 6116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.895205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.895214 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.895266 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.899255 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.900344 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.900351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.900354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.900525 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.901027 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.901035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.901038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.901517 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.905985 (+ 4468us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.906221 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.906573 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":216,"threads_started":1}
I20260619 16:56:06.908442 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.877062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.16.48.3:49211)
0619 16:56:06.888699 (+ 11637us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:06.888721 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:06.888901 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:06.889432 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:06.889437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:06.890168 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:06.890944 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.890957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.892497 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.892500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.893702 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.893716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.895615 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.895622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.895811 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.896474 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:06.896503 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:06.899462 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:06.904293 (+ 4831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.904302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.904308 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.905090 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.905774 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.905782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.905788 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.906001 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.906845 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:06.906854 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:06.907079 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 16:56:06.907239 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 16:56:06.907728 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11443,"thread_start_us":11281,"threads_started":1}
I20260619 16:56:06.910880 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.877190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:49211 (local address 127.16.48.61:38415)
0619 16:56:06.888074 (+ 10884us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.888080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.888793 (+ 713us) server_negotiation.cc:407] Connection header received
0619 16:56:06.889047 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.889053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.889143 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.889222 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.891166 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.892317 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.893876 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.895152 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.895309 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.899694 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.899701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.899706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.899750 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.904072 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.905354 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.905363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.905369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.905584 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.906171 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.906179 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.906185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.906665 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.909722 (+ 3057us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.909913 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.910264 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10747,"thread_start_us":10913,"threads_started":1}
I20260619 16:56:06.905843 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.875485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.3:34353)
0619 16:56:06.876328 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:06.876369 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:06.879390 (+ 3021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:06.880872 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:06.880889 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:06.881913 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:06.883009 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.883043 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.885224 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.885232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.888069 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.888090 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.889465 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.889472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.889890 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.890736 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:06.890778 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:06.894857 (+ 4079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:06.899437 (+ 4580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.899449 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.899469 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.900172 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.900672 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.900676 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.900680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.900882 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.901792 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:06.901806 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:06.902430 (+ 624us) client_negotiation.cc:769] Sending connection context
0619 16:56:06.902863 (+ 433us) client_negotiation.cc:240] Negotiation successful
0619 16:56:06.903653 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":282,"threads_started":1}
I20260619 16:56:06.913093 23992 heartbeater.cc:347] Connected to a master server at 127.16.48.62:41377
I20260619 16:56:06.913532 23992 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:06.914666 23992 heartbeater.cc:511] Master 127.16.48.62:41377 requested a full tablet report, sending...
I20260619 16:56:06.913544 23993 heartbeater.cc:347] Connected to a master server at 127.16.48.61:38415
I20260619 16:56:06.916564 23993 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:06.917071 22730 ts_manager.cc:194] Re-registered known tserver with Master: 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621)
I20260619 16:56:06.917120 23993 heartbeater.cc:511] Master 127.16.48.61:38415 requested a full tablet report, sending...
I20260619 16:56:06.918221 22730 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:34353
Jun 19 16:56:06 dist-test-slave-3ffd krb5kdc[22685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888166, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.60@KRBTEST.COM
I20260619 16:56:06.918427 23331 ts_manager.cc:194] Registered new tserver with Master: 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621)
I20260619 16:56:06.933756 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.893860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:37155 (local address 127.16.48.3:54879)
0619 16:56:06.898631 (+ 4771us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:06.898653 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:06.899613 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:06.900252 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:06.900259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:06.908257 (+ 7998us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:06.909168 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.909188 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.911257 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.911263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.912909 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.912925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.914412 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.914427 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.914742 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.915553 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:06.915587 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:06.923815 (+ 8228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:06.929531 (+ 5716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.929540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.929547 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.930357 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.930952 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:06.930961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:06.930967 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:06.931198 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:06.932103 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:06.932111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:06.932377 (+ 266us) client_negotiation.cc:769] Sending connection context
0619 16:56:06.932549 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 16:56:06.933138 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4625,"thread_start_us":4457,"threads_started":1}
I20260619 16:56:06.935031 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.889151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:54879 (local address 127.16.48.60:37155)
0619 16:56:06.893834 (+ 4683us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.893840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.899174 (+ 5334us) server_negotiation.cc:407] Connection header received
0619 16:56:06.899763 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.899769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.899864 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.899962 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.909466 (+ 9504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.911041 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.913113 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.913903 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.914072 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.924182 (+ 10110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.924192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.924202 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.924261 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.929282 (+ 5021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.930558 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.930568 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.930575 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.930785 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.931351 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.931359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.931364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.931842 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.932023 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.934084 (+ 2061us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.934466 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4552,"thread_start_us":4424,"threads_started":1}
I20260619 16:56:06.936441 23991 heartbeater.cc:347] Connected to a master server at 127.16.48.60:37155
I20260619 16:56:06.936610 23991 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:06.937062 23991 heartbeater.cc:511] Master 127.16.48.60:37155 requested a full tablet report, sending...
I20260619 16:56:06.937083 23995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.901242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.16.48.62:41377)
0619 16:56:06.907445 (+ 6203us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.907453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.913242 (+ 5789us) server_negotiation.cc:407] Connection header received
0619 16:56:06.913519 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.913527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.913642 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.913750 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.919353 (+ 5603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.920610 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.923249 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.923901 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.924015 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.928445 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.928453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.928458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.928518 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.932742 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.933770 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.933776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.933780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.934254 (+ 474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.934781 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.934788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.934795 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.935464 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.935711 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.936379 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.936716 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6085,"thread_start_us":9742,"threads_started":1}
I20260619 16:56:06.938060 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.912894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.16.48.61:38415)
0619 16:56:06.913067 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.913075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.918404 (+ 5329us) server_negotiation.cc:407] Connection header received
0619 16:56:06.918680 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.918687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.918766 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.918832 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.921543 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.923061 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.924995 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.925748 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.925861 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.930482 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.930490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.930499 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.930555 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.934692 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.935615 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.935621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.935626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.935786 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.936289 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.936294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.936297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.936643 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.936896 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.937496 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.937723 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:56:06.938640 23526 ts_manager.cc:194] Registered new tserver with Master: 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621)
I20260619 16:56:06.944183 22730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:56:06.954041 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.918885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.16.48.60:37155)
0619 16:56:06.927675 (+ 8790us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.927681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.938118 (+ 10437us) server_negotiation.cc:407] Connection header received
0619 16:56:06.938183 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.938190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.938272 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.938383 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:06.940881 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.942025 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.943843 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.944509 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.944678 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.947298 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:06.947307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:06.947312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:06.947357 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:06.951289 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.952002 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.952008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.952012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.952201 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:06.952568 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:06.952573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:06.952577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:06.952927 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:06.953149 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.953421 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.953660 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8623,"thread_start_us":8457,"threads_started":1}
W20260619 16:56:06.954396 22730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:56:06.988214 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.977217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.16.48.1:39311)
0619 16:56:06.979626 (+ 2409us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.979636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.979665 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:56:06.979768 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.979774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.979990 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.980233 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:06.981412 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.982663 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.985333 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.986732 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.986878 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.987130 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.987574 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.987833 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2233,"thread_start_us":2220,"threads_started":1}
I20260619 16:56:06.988368 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.976912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:39311 (local address 127.0.0.1:40470)
0619 16:56:06.977528 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:06.977556 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:06.977736 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:06.980225 (+ 2489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:06.980233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:06.980273 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:06.981222 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.981235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.983169 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.983178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.985137 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.985150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.986991 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.987001 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.987310 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.987372 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 16:56:06.987489 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:56:06.987737 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":47,"thread_start_us":323,"threads_started":1}
I20260619 16:56:06.992350 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.977807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.16.48.2:40595)
0619 16:56:06.978685 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.978694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.981360 (+ 2666us) server_negotiation.cc:407] Connection header received
0619 16:56:06.982499 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.982508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.982809 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.983035 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:06.983847 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.986037 (+ 2190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.989106 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.990809 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.990961 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.991463 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.991853 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.992025 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":269,"threads_started":1}
I20260619 16:56:06.992427 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.977607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:40595 (local address 127.0.0.1:59272)
0619 16:56:06.981157 (+ 3550us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:06.981191 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:06.982078 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:06.983013 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:06.983018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:06.983046 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:06.983627 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.983642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.986221 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.986227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.988939 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.988960 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.991340 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.991347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.991664 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.991695 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:56:06.991804 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:56:06.991945 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3393,"mutex_wait_us":178,"thread_start_us":2807,"threads_started":1}
I20260619 16:56:06.998625 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.981470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.16.48.3:33621)
0619 16:56:06.982372 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:06.982384 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:06.989340 (+ 6956us) server_negotiation.cc:407] Connection header received
0619 16:56:06.989808 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:06.989821 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:06.990219 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:06.991089 (+ 870us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:06.991605 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.993702 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.995842 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.997491 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.997665 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.997905 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:06.998131 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 16:56:06.998311 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":299,"threads_started":1}
I20260619 16:56:06.998610 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:06.981219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:33621 (local address 127.0.0.1:34454)
0619 16:56:06.988688 (+ 7469us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:06.988714 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:06.988869 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:06.991083 (+ 2214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:06.991088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:06.991120 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:06.991387 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.991398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.993858 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.993867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.995651 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:06.995665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:06.997712 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:06.997719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:06.997910 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:06.997952 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:56:06.998083 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 16:56:06.998240 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7312,"mutex_wait_us":1,"thread_start_us":12227,"threads_started":1}
I20260619 16:56:06.998100 23785 tablet_service.cc:1467] Processing CreateTablet for tablet 04279ef658df4cf8bded243c8c8b91a9 (DEFAULT_TABLE table=test-table [id=ac2063d77cf547d3ba9513b64c899193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:06.999560 23785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04279ef658df4cf8bded243c8c8b91a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:07.002512 23640 tablet_service.cc:1467] Processing CreateTablet for tablet 04279ef658df4cf8bded243c8c8b91a9 (DEFAULT_TABLE table=test-table [id=ac2063d77cf547d3ba9513b64c899193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:07.002938 23926 tablet_service.cc:1467] Processing CreateTablet for tablet 04279ef658df4cf8bded243c8c8b91a9 (DEFAULT_TABLE table=test-table [id=ac2063d77cf547d3ba9513b64c899193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:07.004493 23640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04279ef658df4cf8bded243c8c8b91a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:07.005995 23926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04279ef658df4cf8bded243c8c8b91a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:07.016923 24027 tablet_bootstrap.cc:492] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: Bootstrap starting.
I20260619 16:56:07.022629 24027 tablet_bootstrap.cc:654] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:07.022805 24028 tablet_bootstrap.cc:492] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: Bootstrap starting.
I20260619 16:56:07.023440 24029 tablet_bootstrap.cc:492] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: Bootstrap starting.
I20260619 16:56:07.024462 24027 log.cc:824] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:07.026862 24029 tablet_bootstrap.cc:654] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:07.027040 24028 tablet_bootstrap.cc:654] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:07.027944 24029 log.cc:824] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:07.029134 24028 log.cc:824] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:07.031309 24029 tablet_bootstrap.cc:492] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: No bootstrap required, opened a new log
I20260619 16:56:07.031656 24029 ts_tablet_manager.cc:1397] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:56:07.032518 24028 tablet_bootstrap.cc:492] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: No bootstrap required, opened a new log
I20260619 16:56:07.032832 24028 ts_tablet_manager.cc:1397] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260619 16:56:07.038537 24027 tablet_bootstrap.cc:492] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: No bootstrap required, opened a new log
I20260619 16:56:07.038921 24027 ts_tablet_manager.cc:1397] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.003s
I20260619 16:56:07.039963 24029 raft_consensus.cc:348] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:07.040434 24029 raft_consensus.cc:374] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:07.040534 24029 raft_consensus.cc:729] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d29389ddcd3472c9b6ea7e1cba2078e, State: Initialized, Role: FOLLOWER
I20260619 16:56:07.041217 24029 consensus_queue.cc:260] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:07.042618 24028 raft_consensus.cc:348] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:07.043095 24028 raft_consensus.cc:374] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:07.043200 24028 raft_consensus.cc:729] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7552fdd23713435ab989a0150f76e121, State: Initialized, Role: FOLLOWER
I20260619 16:56:07.043871 24028 consensus_queue.cc:260] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:07.049876 24028 ts_tablet_manager.cc:1428] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20260619 16:56:07.053452 24029 ts_tablet_manager.cc:1428] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: Time spent starting tablet: real 0.022s user 0.008s sys 0.011s
I20260619 16:56:07.053162 24027 raft_consensus.cc:348] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:07.053718 24027 raft_consensus.cc:374] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:07.053830 24027 raft_consensus.cc:729] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731e6bc712914e86a3c5307622e64f51, State: Initialized, Role: FOLLOWER
I20260619 16:56:07.054057 23992 heartbeater.cc:503] Master 127.16.48.62:41377 was elected leader, sending a full tablet report...
I20260619 16:56:07.054673 24027 consensus_queue.cc:260] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:07.058815 23851 heartbeater.cc:503] Master 127.16.48.62:41377 was elected leader, sending a full tablet report...
I20260619 16:56:07.059770 24027 ts_tablet_manager.cc:1428] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: Time spent starting tablet: real 0.021s user 0.008s sys 0.010s
I20260619 16:56:07.063462 24035 raft_consensus.cc:482] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:56:07.063781 24035 raft_consensus.cc:504] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:07.065359 24035 leader_election.cc:288] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311), 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621)
I20260619 16:56:07.074163 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:07.065698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:53715 (local address 127.16.48.1:39311)
0619 16:56:07.065891 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:07.065895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:07.066543 (+ 648us) server_negotiation.cc:407] Connection header received
0619 16:56:07.066696 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:07.066701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:07.066792 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:07.066898 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:07.067688 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.068758 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.071800 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.073092 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.073239 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:07.073349 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:07.073777 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 16:56:07.073878 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:56:07.074345 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:07.065701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:39311 (local address 127.16.48.2:53715)
0619 16:56:07.066406 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:07.066425 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:07.066556 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:07.067066 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:07.067071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:07.067096 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:07.067500 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:07.067510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.068880 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.068884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:07.071592 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:07.071613 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.073383 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.073389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:07.073558 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:07.073610 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 16:56:07.073748 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:56:07.073898 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":1,"thread_start_us":368,"threads_started":1}
I20260619 16:56:07.074390 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:07.066648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:53059 (local address 127.16.48.3:33621)
0619 16:56:07.066883 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:07.066889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:07.067267 (+ 378us) server_negotiation.cc:407] Connection header received
0619 16:56:07.067423 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:07.067427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:07.067480 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:07.067549 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:07.068079 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.069081 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.071841 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.073092 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.073239 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:07.073349 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:07.074021 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 16:56:07.074152 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:56:07.074450 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:07.066488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:33621 (local address 127.16.48.2:53059)
0619 16:56:07.067162 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:07.067180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:07.067309 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:07.067591 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:07.067595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:07.067620 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:07.067900 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:07.067908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.069203 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.069208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:07.071690 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:07.071704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.073682 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.073691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:07.073858 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:07.073888 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:56:07.073973 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:56:07.074110 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":374,"threads_started":1}
I20260619 16:56:07.075471 23946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04279ef658df4cf8bded243c8c8b91a9" candidate_uuid: "731e6bc712914e86a3c5307622e64f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" is_pre_election: true
I20260619 16:56:07.075511 23660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04279ef658df4cf8bded243c8c8b91a9" candidate_uuid: "731e6bc712914e86a3c5307622e64f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7552fdd23713435ab989a0150f76e121" is_pre_election: true
I20260619 16:56:07.075999 23660 raft_consensus.cc:2454] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731e6bc712914e86a3c5307622e64f51 in term 0.
I20260619 16:56:07.075996 23946 raft_consensus.cc:2454] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731e6bc712914e86a3c5307622e64f51 in term 0.
I20260619 16:56:07.076761 23738 leader_election.cc:302] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731e6bc712914e86a3c5307622e64f51, 7552fdd23713435ab989a0150f76e121; no voters:
I20260619 16:56:07.077189 24035 raft_consensus.cc:2781] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:56:07.077353 24035 raft_consensus.cc:482] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:56:07.077453 24035 raft_consensus.cc:3037] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:07.079803 24035 raft_consensus.cc:504] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:07.080494 24035 leader_election.cc:288] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [CANDIDATE]: Term 1 election: Requested vote from peers 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311), 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621)
I20260619 16:56:07.081058 23660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04279ef658df4cf8bded243c8c8b91a9" candidate_uuid: "731e6bc712914e86a3c5307622e64f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7552fdd23713435ab989a0150f76e121"
I20260619 16:56:07.081073 23946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04279ef658df4cf8bded243c8c8b91a9" candidate_uuid: "731e6bc712914e86a3c5307622e64f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e"
I20260619 16:56:07.081296 23660 raft_consensus.cc:3037] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:07.081316 23946 raft_consensus.cc:3037] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:07.083765 23660 raft_consensus.cc:2454] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731e6bc712914e86a3c5307622e64f51 in term 1.
I20260619 16:56:07.084267 23946 raft_consensus.cc:2454] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731e6bc712914e86a3c5307622e64f51 in term 1.
I20260619 16:56:07.084311 23738 leader_election.cc:302] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731e6bc712914e86a3c5307622e64f51, 7552fdd23713435ab989a0150f76e121; no voters:
I20260619 16:56:07.084676 24035 raft_consensus.cc:2781] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:07.085125 24035 raft_consensus.cc:686] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 1 LEADER]: Becoming Leader. State: Replica: 731e6bc712914e86a3c5307622e64f51, State: Running, Role: LEADER
I20260619 16:56:07.085703 24035 consensus_queue.cc:237] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:07.095939 22730 catalog_manager.cc:5495] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731e6bc712914e86a3c5307622e64f51 (127.16.48.2). New cstate: current_term: 1 leader_uuid: "731e6bc712914e86a3c5307622e64f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } health_report { overall_health: UNKNOWN } } }
W20260619 16:56:07.110387 23713 tablet.cc:2369] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:56:07.119733 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:07.113816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.16.48.2:40595)
0619 16:56:07.114046 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:07.114052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:07.114167 (+ 115us) server_negotiation.cc:407] Connection header received
0619 16:56:07.114436 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:07.114441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:07.114533 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:07.114724 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:07.115132 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.115834 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.117386 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.117810 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.117949 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:07.118490 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:07.119050 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:07.119227 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:07.119313 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 16:56:07.119399 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 16:56:07.125845 23994 tablet.cc:2369] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:56:07.129226 23660 raft_consensus.cc:1261] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [term 1 FOLLOWER]: Refusing update from remote peer 731e6bc712914e86a3c5307622e64f51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:07.129226 23946 raft_consensus.cc:1261] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 1 FOLLOWER]: Refusing update from remote peer 731e6bc712914e86a3c5307622e64f51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:07.130335 24038 consensus_queue.cc:1035] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:07.130752 24035 consensus_queue.cc:1035] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:07.154933 24043 mvcc.cc:205] Tried to move back new op lower bound from 7298613932549226496 to 7298613932386750464. Current Snapshot: MvccSnapshot[applied={T|T < 7298613932549226496}]
I20260619 16:56:07.174613 22729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:07.175017 22729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:07.183661 22729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7789317184b442768fe9397c633f7ee6: Sending DeleteTablet for 3 replicas of tablet 04279ef658df4cf8bded243c8c8b91a9
I20260619 16:56:07.184842 23640 tablet_service.cc:1514] Processing DeleteTablet for tablet 04279ef658df4cf8bded243c8c8b91a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:07 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:40470
I20260619 16:56:07.185246 23785 tablet_service.cc:1514] Processing DeleteTablet for tablet 04279ef658df4cf8bded243c8c8b91a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:07 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:59272
I20260619 16:56:07.185305 23926 tablet_service.cc:1514] Processing DeleteTablet for tablet 04279ef658df4cf8bded243c8c8b91a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:07 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:34454
I20260619 16:56:07.199051 24052 tablet_replica.cc:330] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: stopping tablet replica
I20260619 16:56:07.199867 24053 tablet_replica.cc:330] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: stopping tablet replica
I20260619 16:56:07.200037 24052 raft_consensus.cc:2229] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:07.200595 24053 raft_consensus.cc:2229] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:56:07.200788 24052 raft_consensus.cc:2258] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:07.201526 24053 raft_consensus.cc:2258] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:07.202687 24051 tablet_replica.cc:330] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: stopping tablet replica
I20260619 16:56:07.203280 24051 raft_consensus.cc:2229] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:07.203495 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 22698
I20260619 16:56:07.204020 24051 raft_consensus.cc:2258] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:07.204820 22760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:07.205091 24053 ts_tablet_manager.cc:1905] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:07.206694 24052 ts_tablet_manager.cc:1905] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:07.208066 24051 ts_tablet_manager.cc:1905] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:07.216442 24051 ts_tablet_manager.cc:1918] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:07.216660 24051 log.cc:1196] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/04279ef658df4cf8bded243c8c8b91a9
I20260619 16:56:07.217270 24051 ts_tablet_manager.cc:1939] T 04279ef658df4cf8bded243c8c8b91a9 P 7552fdd23713435ab989a0150f76e121: Deleting consensus metadata
I20260619 16:56:07.223181 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:07.188229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.16.48.61:38415)
0619 16:56:07.188415 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:07.188422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:07.201889 (+ 13467us) server_negotiation.cc:407] Connection header received
0619 16:56:07.202708 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:07.202718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:07.202812 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:07.202893 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:07.207154 (+ 4261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.208385 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.210582 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.211252 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.211364 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:07.215848 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:07.215855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:07.215860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:07.215910 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:07.219660 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:07.220717 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:07.220724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:07.220728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:07.220881 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:07.221309 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:07.221314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:07.221318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:07.221673 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:07.221928 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:07.222568 (+ 640us) server_negotiation.cc:299] Negotiation successful
0619 16:56:07.222813 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:56:07.225997 24052 ts_tablet_manager.cc:1918] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:07.226392 24052 log.cc:1196] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/04279ef658df4cf8bded243c8c8b91a9
I20260619 16:56:07.227248 24052 ts_tablet_manager.cc:1939] T 04279ef658df4cf8bded243c8c8b91a9 P 2d29389ddcd3472c9b6ea7e1cba2078e: Deleting consensus metadata
I20260619 16:56:07.236860 24053 ts_tablet_manager.cc:1918] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:07.237082 24053 log.cc:1196] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/04279ef658df4cf8bded243c8c8b91a9
I20260619 16:56:07.237702 24053 ts_tablet_manager.cc:1939] T 04279ef658df4cf8bded243c8c8b91a9 P 731e6bc712914e86a3c5307622e64f51: Deleting consensus metadata
I20260619 16:56:07.242865 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:07.202102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.16.48.60:37155)
0619 16:56:07.202308 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:07.202314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:07.224176 (+ 21862us) server_negotiation.cc:407] Connection header received
0619 16:56:07.224418 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:07.224424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:07.224507 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:07.224580 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:07.226999 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.228087 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.229999 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.230657 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.230749 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:07.234946 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:07.234953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:07.234959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:07.235015 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:07.239056 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:07.240242 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:07.240250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:07.240256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:07.240621 (+ 365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:07.241172 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:07.241177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:07.241181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:07.241529 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:07.241753 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:07.242288 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 16:56:07.242523 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:56:07.363564 22714 catalog_manager.cc:4841] TS 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621): tablet 04279ef658df4cf8bded243c8c8b91a9 (table test-table [id=ac2063d77cf547d3ba9513b64c899193]) successfully deleted
I20260619 16:56:07.363947 22714 catalog_manager.cc:4841] TS 731e6bc712914e86a3c5307622e64f51 (127.16.48.2:40595): tablet 04279ef658df4cf8bded243c8c8b91a9 (table test-table [id=ac2063d77cf547d3ba9513b64c899193]) successfully deleted
I20260619 16:56:07.364158 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 22698
I20260619 16:56:07.364187 22714 catalog_manager.cc:4841] TS 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311): tablet 04279ef658df4cf8bded243c8c8b91a9 (table test-table [id=ac2063d77cf547d3ba9513b64c899193]) successfully deleted
I20260619 16:56:07.418860 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:07.401486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.16.48.61:38415)
0619 16:56:07.401740 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:07.401747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:07.401771 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:56:07.401980 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:07.401999 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:07.402087 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:07.402170 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:07.404931 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.406101 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.407662 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.408221 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.408366 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:07.411686 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:07.411697 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:07.411707 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:07.411745 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:07.415507 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:07.416614 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:07.416621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:07.416624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:07.416787 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:07.417260 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:07.417265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:07.417270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:07.417585 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:07.417774 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:07.418271 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 16:56:07.418513 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:56:07.420235 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:07.402532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47878 (local address 127.16.48.60:37155)
0619 16:56:07.402719 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:07.402725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:07.402955 (+ 230us) server_negotiation.cc:407] Connection header received
0619 16:56:07.403126 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:07.403132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:07.403233 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:07.403365 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:07.405473 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.406253 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.408583 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:07.409216 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:07.409371 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:07.412715 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:07.412724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:07.412731 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:07.412778 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:07.416217 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:07.417019 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:07.417024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:07.417028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:07.417158 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:07.418102 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:07.418108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:07.418111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:07.418466 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:07.418716 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:07.419593 (+ 877us) server_negotiation.cc:299] Negotiation successful
0619 16:56:07.419820 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:56:08.160446 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.159572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:43401)
0619 16:56:08.160067 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.160263 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":277,"threads_started":1}
W20260619 16:56:08.160889 23851 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:41377 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
I20260619 16:56:08.164919 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.163941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.3:49637)
0619 16:56:08.164490 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.164678 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":296,"threads_started":1}
I20260619 16:56:08.165258 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.164402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.1:45085)
0619 16:56:08.164887 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.165034 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":260,"threads_started":1}
W20260619 16:56:08.165618 23992 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:41377 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
W20260619 16:56:08.166373 23706 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:41377 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
I20260619 16:56:08.683269 24065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7789317184b442768fe9397c633f7ee6)
I20260619 16:56:08.683521 24066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7789317184b442768fe9397c633f7ee6)
I20260619 16:56:08.683590 24065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } }
I20260619 16:56:08.683786 24066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } }
I20260619 16:56:08.685214 24066 leader_election.cc:288] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377), 2c609163a3874286809b0d1adc1405da (127.16.48.61:38415)
I20260619 16:56:08.685220 24065 leader_election.cc:288] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377), 693be2ed13624107816f36c365395757 (127.16.48.60:37155)
I20260619 16:56:08.687983 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.685348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33616)
0619 16:56:08.687526 (+ 2178us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.687721 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2027,"thread_start_us":1895,"threads_started":1}
I20260619 16:56:08.688028 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.685344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33618)
0619 16:56:08.687515 (+ 2171us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.687691 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2028,"thread_start_us":1871,"threads_started":1}
W20260619 16:56:08.688860 23511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
W20260619 16:56:08.692837 23317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
I20260619 16:56:08.698592 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.687678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:37155 (local address 127.0.0.1:47882)
0619 16:56:08.688365 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.688394 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:08.688591 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:08.689602 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:08.689611 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:08.689661 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:08.690028 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:08.690045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:08.692724 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:08.692729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:08.695362 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:08.695379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:08.697490 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:08.697499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:08.697780 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:08.697831 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 16:56:08.697928 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:56:08.698107 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":5,"thread_start_us":4794,"threads_started":1}
I20260619 16:56:08.698997 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.688548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.0.0.1:59218)
0619 16:56:08.689004 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.689028 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:08.689164 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:08.689864 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:08.689872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:08.689908 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:08.690182 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:08.690198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:08.692385 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:08.692394 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:08.695282 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:08.695310 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:08.697863 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:08.697873 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:08.698174 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:08.698235 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 16:56:08.698341 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:56:08.698571 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326}
I20260619 16:56:08.699690 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.688184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.16.48.60:37155)
0619 16:56:08.689178 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:08.689186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:08.689212 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:56:08.689295 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:08.689302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:08.689394 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:08.690205 (+ 811us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:08.690564 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:08.692209 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:08.695607 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:08.697209 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:08.698886 (+ 1677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:08.699030 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:08.699089 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 16:56:08.699255 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":692,"threads_started":1}
I20260619 16:56:08.700163 24070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.687756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.16.48.61:38415)
0619 16:56:08.688833 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:08.688841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:08.689277 (+ 436us) server_negotiation.cc:407] Connection header received
0619 16:56:08.689341 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:08.689347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:08.689427 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:08.689563 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:08.690391 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:08.692140 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:08.695581 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:08.697511 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:08.699253 (+ 1742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:08.699418 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:08.699485 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 16:56:08.699694 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1454,"server-negotiator.queue_time_us":951,"thread_start_us":1312,"threads_started":1}
I20260619 16:56:08.701751 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "693be2ed13624107816f36c365395757" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c609163a3874286809b0d1adc1405da" is_pre_election: true
I20260619 16:56:08.702198 23341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 693be2ed13624107816f36c365395757 in term 1.
I20260619 16:56:08.701550 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c609163a3874286809b0d1adc1405da" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "693be2ed13624107816f36c365395757" is_pre_election: true
I20260619 16:56:08.701776 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.701353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33644)
0619 16:56:08.701515 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.701610 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 16:56:08.702768 23536 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c609163a3874286809b0d1adc1405da in term 1.
I20260619 16:56:08.702804 23511 leader_election.cc:302] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c609163a3874286809b0d1adc1405da, 693be2ed13624107816f36c365395757; no voters:
W20260619 16:56:08.702844 23317 leader_election.cc:334] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377): Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
I20260619 16:56:08.703207 24066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:56:08.703322 24066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7789317184b442768fe9397c633f7ee6)
I20260619 16:56:08.703413 24066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:56:08.703620 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.702487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33660)
0619 16:56:08.703264 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.703368 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640}
I20260619 16:56:08.703790 23317 leader_election.cc:302] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c609163a3874286809b0d1adc1405da, 693be2ed13624107816f36c365395757; no voters: 7789317184b442768fe9397c633f7ee6
W20260619 16:56:08.703855 23511 leader_election.cc:334] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377): Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
I20260619 16:56:08.704552 24065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:56:08.704679 24065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7789317184b442768fe9397c633f7ee6)
I20260619 16:56:08.704782 24065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:56:08.706611 24066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } }
I20260619 16:56:08.707346 24066 leader_election.cc:288] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 2 election: Requested vote from peers 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377), 2c609163a3874286809b0d1adc1405da (127.16.48.61:38415)
I20260619 16:56:08.707696 24065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } }
I20260619 16:56:08.707931 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.707503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33664)
0619 16:56:08.707678 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.707779 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:56:08.708192 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "693be2ed13624107816f36c365395757" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c609163a3874286809b0d1adc1405da"
I20260619 16:56:08.708515 23341 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 693be2ed13624107816f36c365395757 in current term 2: Already voted for candidate 2c609163a3874286809b0d1adc1405da in this term.
I20260619 16:56:08.708518 24065 leader_election.cc:288] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [CANDIDATE]: Term 2 election: Requested vote from peers 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377), 693be2ed13624107816f36c365395757 (127.16.48.60:37155)
I20260619 16:56:08.709157 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.708633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33674)
0619 16:56:08.708901 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.708989 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260619 16:56:08.709321 23511 leader_election.cc:334] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377): Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
I20260619 16:56:08.709456 23511 leader_election.cc:302] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 693be2ed13624107816f36c365395757; no voters: 2c609163a3874286809b0d1adc1405da, 7789317184b442768fe9397c633f7ee6
I20260619 16:56:08.709857 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.709351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33672)
0619 16:56:08.709597 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.709694 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 16:56:08.709851 24066 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 16:56:08.709985 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c609163a3874286809b0d1adc1405da" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "693be2ed13624107816f36c365395757"
I20260619 16:56:08.710327 23536 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c609163a3874286809b0d1adc1405da in current term 2: Already voted for candidate 693be2ed13624107816f36c365395757 in this term.
I20260619 16:56:08.711082 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:08.710633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33678)
0619 16:56:08.710859 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:08.710942 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 16:56:08.711346 23317 leader_election.cc:334] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377): Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
I20260619 16:56:08.711503 23317 leader_election.cc:302] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c609163a3874286809b0d1adc1405da; no voters: 693be2ed13624107816f36c365395757, 7789317184b442768fe9397c633f7ee6
I20260619 16:56:08.711792 24065 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 16:56:09.162385 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:09.161640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:46895)
0619 16:56:09.162105 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:09.162203 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":231,"threads_started":1}
I20260619 16:56:09.167878 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:09.166644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.3:49659)
0619 16:56:09.167206 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:09.167660 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":311,"threads_started":1}
I20260619 16:56:09.167912 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:09.167099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.1:50899)
0619 16:56:09.167580 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:09.167679 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":288,"threads_started":1}
I20260619 16:56:10.163949 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.163221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:33781)
0619 16:56:10.163666 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.163762 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260619 16:56:10.169867 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.168988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.3:38931)
0619 16:56:10.169516 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.169614 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":299,"threads_started":1}
I20260619 16:56:10.170881 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.169578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.1:51307)
0619 16:56:10.170593 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.170689 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":814,"threads_started":1}
I20260619 16:56:10.431545 24080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:56:10.431761 24080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } }
I20260619 16:56:10.432554 24080 leader_election.cc:288] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377), 2c609163a3874286809b0d1adc1405da (127.16.48.61:38415)
I20260619 16:56:10.433823 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.432849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33932)
0619 16:56:10.433401 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.433540 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":283,"threads_started":1}
I20260619 16:56:10.433919 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "693be2ed13624107816f36c365395757" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c609163a3874286809b0d1adc1405da" is_pre_election: true
I20260619 16:56:10.434223 23341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 693be2ed13624107816f36c365395757 in term 2.
I20260619 16:56:10.434754 23511 leader_election.cc:302] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c609163a3874286809b0d1adc1405da, 693be2ed13624107816f36c365395757; no voters:
I20260619 16:56:10.435055 24080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 16:56:10.435166 24080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:56:10.435286 24080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:56:10.435632 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.435231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33936)
0619 16:56:10.435398 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.435503 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:56:10.435834 23511 leader_election.cc:334] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377): Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
I20260619 16:56:10.437608 24080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } }
I20260619 16:56:10.438187 24080 leader_election.cc:288] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 3 election: Requested vote from peers 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377), 2c609163a3874286809b0d1adc1405da (127.16.48.61:38415)
I20260619 16:56:10.438874 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.438440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33940)
0619 16:56:10.438599 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.438708 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 16:56:10.439122 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "693be2ed13624107816f36c365395757" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c609163a3874286809b0d1adc1405da"
I20260619 16:56:10.439356 23341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:56:10.439922 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.439575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33950)
0619 16:56:10.439719 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.439795 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 16:56:10.440158 23511 leader_election.cc:334] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377): Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
I20260619 16:56:10.441797 23341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 693be2ed13624107816f36c365395757 in term 3.
I20260619 16:56:10.442308 23511 leader_election.cc:302] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c609163a3874286809b0d1adc1405da, 693be2ed13624107816f36c365395757; no voters: 7789317184b442768fe9397c633f7ee6
I20260619 16:56:10.442689 24080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:56:10.443161 24080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [term 3 LEADER]: Becoming Leader. State: Replica: 693be2ed13624107816f36c365395757, State: Running, Role: LEADER
I20260619 16:56:10.443562 24080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } }
I20260619 16:56:10.445278 24083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693be2ed13624107816f36c365395757. Latest consensus state: current_term: 3 leader_uuid: "693be2ed13624107816f36c365395757" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:10.445571 24083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:10.446244 24085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:56:10.450030 24085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:56:10.450953 24085 catalog_manager.cc:1259] Loaded cluster ID: b3fe076130634d9a832f2c816f6e384e
I20260619 16:56:10.451079 24085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:56:10.452359 24085 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:56:10.453193 24085 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Loaded TSK: 0
I20260619 16:56:10.454107 24085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:56:10.481832 23526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:56:10.489192 23526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:56:10.494272 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.493720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33978)
0619 16:56:10.493959 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.494071 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 16:56:10.494249 23341 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [term 3 FOLLOWER]: Refusing update from remote peer 693be2ed13624107816f36c365395757: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 16:56:10.495306 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.494915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33990)
0619 16:56:10.495106 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.495184 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 16:56:10.495301 24083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 16:56:10.495779 23511 consensus_peers.cc:487] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 -> Peer 7789317184b442768fe9397c633f7ee6 (127.16.48.62:41377): Couldn't send request to peer 7789317184b442768fe9397c633f7ee6. Status: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:56:10.499125 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693be2ed13624107816f36c365395757. Latest consensus state: current_term: 3 leader_uuid: "693be2ed13624107816f36c365395757" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:10.499464 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:56:10.500831 24080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "693be2ed13624107816f36c365395757" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:10.501202 24080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:10.501986 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "693be2ed13624107816f36c365395757" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7789317184b442768fe9397c633f7ee6" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 41377 } } peers { permanent_uuid: "2c609163a3874286809b0d1adc1405da" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "693be2ed13624107816f36c365395757" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 37155 } attrs { promote: false } } }
I20260619 16:56:10.504694 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c609163a3874286809b0d1adc1405da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:56:10.525790 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.510891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:39311 (local address 127.0.0.1:42740)
0619 16:56:10.511456 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.511510 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:10.511655 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:10.519733 (+ 8078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:10.519738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:10.519766 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:10.520316 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:10.520332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.521745 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.521750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:10.523618 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:10.523632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.524902 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.524911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:10.525087 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.525111 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:56:10.525244 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 16:56:10.525388 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":296,"threads_started":1}
I20260619 16:56:10.525765 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.511374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.16.48.1:39311)
0619 16:56:10.519383 (+ 8009us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:10.519387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:10.519403 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:56:10.519475 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:10.519480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:10.519602 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:10.519717 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:10.520575 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.521597 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.523801 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.524690 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.524840 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.524960 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:10.525270 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:56:10.525407 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7898,"thread_start_us":7810,"threads_started":1}
I20260619 16:56:10.527011 23640 tablet_service.cc:1467] Processing CreateTablet for tablet 412d7b71e7fe44738c279b869315eef6 (DEFAULT_TABLE table=test-table [id=97785fe80a444d0db6b6ebfe89e76d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:10.527522 23640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412d7b71e7fe44738c279b869315eef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:10.528605 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.510455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:40595 (local address 127.0.0.1:50918)
0619 16:56:10.510636 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.510657 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:10.510790 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:10.520698 (+ 9908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:10.520702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:10.520729 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:10.521065 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:10.521077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.522524 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.522529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:10.526077 (+ 3548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:10.526092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.527749 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.527755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:10.527940 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.527981 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:56:10.528052 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:56:10.528249 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":175}
I20260619 16:56:10.529093 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.510531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.16.48.2:40595)
0619 16:56:10.520329 (+ 9798us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:10.520335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:10.520355 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:56:10.520430 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:10.520437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:10.520529 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:10.520651 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:10.521289 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.522324 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.526283 (+ 3959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.527535 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.528289 (+ 754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.528409 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:10.528572 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 16:56:10.528743 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9653,"thread_start_us":9522,"threads_started":1}
I20260619 16:56:10.529320 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.511873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.16.48.3:33621)
0619 16:56:10.512481 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:10.512491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:10.518797 (+ 6306us) server_negotiation.cc:407] Connection header received
0619 16:56:10.519066 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:10.519076 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:10.519191 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:10.519324 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:10.520859 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.522085 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.526788 (+ 4703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.528089 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.528234 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.528340 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:10.528895 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 16:56:10.529029 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":291,"threads_started":1}
I20260619 16:56:10.529856 23785 tablet_service.cc:1467] Processing CreateTablet for tablet 412d7b71e7fe44738c279b869315eef6 (DEFAULT_TABLE table=test-table [id=97785fe80a444d0db6b6ebfe89e76d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:10.530019 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.511527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:33621 (local address 127.0.0.1:47588)
0619 16:56:10.518618 (+ 7091us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.518647 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:10.518844 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:10.519701 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:10.519710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:10.519749 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:10.520333 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:10.520346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.522245 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.522250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:10.526355 (+ 4105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:10.526371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.528486 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.528492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:10.528695 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.528728 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:56:10.529500 (+ 772us) client_negotiation.cc:240] Negotiation successful
0619 16:56:10.529656 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6870,"mutex_wait_us":145,"spinlock_wait_cycles":1280,"thread_start_us":6664,"threads_started":1}
I20260619 16:56:10.530323 23785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412d7b71e7fe44738c279b869315eef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:10.533106 23926 tablet_service.cc:1467] Processing CreateTablet for tablet 412d7b71e7fe44738c279b869315eef6 (DEFAULT_TABLE table=test-table [id=97785fe80a444d0db6b6ebfe89e76d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:10.533615 23926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412d7b71e7fe44738c279b869315eef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:10.537169 24099 tablet_bootstrap.cc:492] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: Bootstrap starting.
I20260619 16:56:10.539706 24100 tablet_bootstrap.cc:492] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: Bootstrap starting.
I20260619 16:56:10.539793 24099 tablet_bootstrap.cc:654] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:10.539955 24098 tablet_bootstrap.cc:492] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: Bootstrap starting.
I20260619 16:56:10.541745 24100 tablet_bootstrap.cc:654] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:10.542418 24098 tablet_bootstrap.cc:654] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:10.543747 24100 tablet_bootstrap.cc:492] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: No bootstrap required, opened a new log
I20260619 16:56:10.543911 24100 ts_tablet_manager.cc:1397] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:56:10.544732 24100 raft_consensus.cc:348] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:10.544932 24098 tablet_bootstrap.cc:492] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: No bootstrap required, opened a new log
I20260619 16:56:10.544958 24100 raft_consensus.cc:374] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:10.545040 24100 raft_consensus.cc:729] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d29389ddcd3472c9b6ea7e1cba2078e, State: Initialized, Role: FOLLOWER
I20260619 16:56:10.545090 24099 tablet_bootstrap.cc:492] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: No bootstrap required, opened a new log
I20260619 16:56:10.545123 24098 ts_tablet_manager.cc:1397] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:56:10.545246 24099 ts_tablet_manager.cc:1397] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 16:56:10.545344 24100 consensus_queue.cc:260] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:10.545970 24098 raft_consensus.cc:348] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:10.546164 24100 ts_tablet_manager.cc:1428] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:56:10.546010 24099 raft_consensus.cc:348] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:10.546278 24099 raft_consensus.cc:374] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:10.546286 24098 raft_consensus.cc:374] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:10.546389 24099 raft_consensus.cc:729] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731e6bc712914e86a3c5307622e64f51, State: Initialized, Role: FOLLOWER
I20260619 16:56:10.546389 24098 raft_consensus.cc:729] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7552fdd23713435ab989a0150f76e121, State: Initialized, Role: FOLLOWER
I20260619 16:56:10.546608 24099 consensus_queue.cc:260] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:10.546656 24098 consensus_queue.cc:260] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:10.547991 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.546862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.3:54047)
0619 16:56:10.547708 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.547817 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":730}
I20260619 16:56:10.548539 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.548007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:49231)
0619 16:56:10.548295 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.548390 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260619 16:56:10.549806 23991 heartbeater.cc:503] Master 127.16.48.60:37155 was elected leader, sending a full tablet report...
I20260619 16:56:10.550756 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.550156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.3:46713)
0619 16:56:10.550404 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.550529 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 16:56:10.552387 24099 ts_tablet_manager.cc:1428] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260619 16:56:10.552767 24098 ts_tablet_manager.cc:1428] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: Time spent starting tablet: real 0.008s user 0.001s sys 0.003s
I20260619 16:56:10.552784 23850 heartbeater.cc:503] Master 127.16.48.60:37155 was elected leader, sending a full tablet report...
I20260619 16:56:10.553826 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.553387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.1:49675)
0619 16:56:10.553536 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.553650 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 16:56:10.555042 23705 heartbeater.cc:503] Master 127.16.48.60:37155 was elected leader, sending a full tablet report...
I20260619 16:56:10.555262 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.554833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:42247)
0619 16:56:10.555000 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.555091 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:56:10.555799 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.554935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.1:45563)
0619 16:56:10.555613 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.555682 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586}
I20260619 16:56:10.573736 24105 raft_consensus.cc:482] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:56:10.573987 24105 raft_consensus.cc:504] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:10.574740 24105 leader_election.cc:288] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311), 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621)
I20260619 16:56:10.575390 23946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d7b71e7fe44738c279b869315eef6" candidate_uuid: "731e6bc712914e86a3c5307622e64f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" is_pre_election: true
I20260619 16:56:10.575383 23660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d7b71e7fe44738c279b869315eef6" candidate_uuid: "731e6bc712914e86a3c5307622e64f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7552fdd23713435ab989a0150f76e121" is_pre_election: true
I20260619 16:56:10.575660 23660 raft_consensus.cc:2454] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731e6bc712914e86a3c5307622e64f51 in term 0.
I20260619 16:56:10.575660 23946 raft_consensus.cc:2454] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731e6bc712914e86a3c5307622e64f51 in term 0.
I20260619 16:56:10.576216 23738 leader_election.cc:302] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731e6bc712914e86a3c5307622e64f51, 7552fdd23713435ab989a0150f76e121; no voters:
I20260619 16:56:10.576496 24105 raft_consensus.cc:2781] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:56:10.576630 24105 raft_consensus.cc:482] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:56:10.576725 24105 raft_consensus.cc:3037] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:10.578781 24105 raft_consensus.cc:504] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:10.579406 24105 leader_election.cc:288] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [CANDIDATE]: Term 1 election: Requested vote from peers 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311), 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621)
I20260619 16:56:10.580082 23660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d7b71e7fe44738c279b869315eef6" candidate_uuid: "731e6bc712914e86a3c5307622e64f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7552fdd23713435ab989a0150f76e121"
I20260619 16:56:10.580099 23946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d7b71e7fe44738c279b869315eef6" candidate_uuid: "731e6bc712914e86a3c5307622e64f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e"
I20260619 16:56:10.580377 23946 raft_consensus.cc:3037] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:10.580364 23660 raft_consensus.cc:3037] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:10.582427 23946 raft_consensus.cc:2454] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731e6bc712914e86a3c5307622e64f51 in term 1.
I20260619 16:56:10.582428 23660 raft_consensus.cc:2454] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731e6bc712914e86a3c5307622e64f51 in term 1.
I20260619 16:56:10.582974 23738 leader_election.cc:302] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d29389ddcd3472c9b6ea7e1cba2078e, 731e6bc712914e86a3c5307622e64f51; no voters:
I20260619 16:56:10.583371 24105 raft_consensus.cc:2781] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:10.583873 24105 raft_consensus.cc:686] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 1 LEADER]: Becoming Leader. State: Replica: 731e6bc712914e86a3c5307622e64f51, State: Running, Role: LEADER
I20260619 16:56:10.584457 24105 consensus_queue.cc:237] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } }
I20260619 16:56:10.585068 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.584668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:55555)
0619 16:56:10.584829 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.584920 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 16:56:10.587615 23526 catalog_manager.cc:5495] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731e6bc712914e86a3c5307622e64f51 (127.16.48.2). New cstate: current_term: 1 leader_uuid: "731e6bc712914e86a3c5307622e64f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e6bc712914e86a3c5307622e64f51" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 } health_report { overall_health: UNKNOWN } } }
I20260619 16:56:10.623410 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.618121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.16.48.2:40595)
0619 16:56:10.618383 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:10.618388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:10.618410 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:56:10.618492 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:10.618496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:10.618569 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:10.618664 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:10.619011 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.619824 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.621434 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.621873 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.621986 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.622409 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:10.622595 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:10.622678 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:10.622885 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 16:56:10.622961 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 16:56:10.626883 23946 raft_consensus.cc:1261] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 1 FOLLOWER]: Refusing update from remote peer 731e6bc712914e86a3c5307622e64f51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:10.626883 23660 raft_consensus.cc:1261] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [term 1 FOLLOWER]: Refusing update from remote peer 731e6bc712914e86a3c5307622e64f51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:10.627590 24107 consensus_queue.cc:1035] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7552fdd23713435ab989a0150f76e121" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:10.628064 24105 consensus_queue.cc:1035] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d29389ddcd3472c9b6ea7e1cba2078e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 33621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:10.630725 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.630256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.1:57493)
0619 16:56:10.630477 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.630555 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 16:56:10.630918 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.630267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.3:48071)
0619 16:56:10.630595 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.630714 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260619 16:56:10.634446 24110 mvcc.cc:205] Tried to move back new op lower bound from 7298613946881007616 to 7298613946716565504. Current Snapshot: MvccSnapshot[applied={T|T < 7298613946881007616}]
I20260619 16:56:10.638410 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.637520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.1:36467)
0619 16:56:10.637991 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.638128 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
I20260619 16:56:10.643649 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.642766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:57003)
0619 16:56:10.643254 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.643387 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313}
I20260619 16:56:10.652133 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.651389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.3:43411)
0619 16:56:10.651744 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.651878 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260619 16:56:10.653967 23525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:10.654384 23525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:10.657999 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.657476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.16.48.2:57051)
0619 16:56:10.657692 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:10.657794 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 16:56:10.662882 23525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757: Sending DeleteTablet for 3 replicas of tablet 412d7b71e7fe44738c279b869315eef6
I20260619 16:56:10.663143 23926 tablet_service.cc:1514] Processing DeleteTablet for tablet 412d7b71e7fe44738c279b869315eef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:10 UTC) from {username='oryx', principal='oryx/127.16.48.60@KRBTEST.COM'} at 127.0.0.1:47588
I20260619 16:56:10.663250 23785 tablet_service.cc:1514] Processing DeleteTablet for tablet 412d7b71e7fe44738c279b869315eef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:10 UTC) from {username='oryx', principal='oryx/127.16.48.60@KRBTEST.COM'} at 127.0.0.1:50918
I20260619 16:56:10.663997 24120 tablet_replica.cc:330] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: stopping tablet replica
I20260619 16:56:10.663959 23640 tablet_service.cc:1514] Processing DeleteTablet for tablet 412d7b71e7fe44738c279b869315eef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:10 UTC) from {username='oryx', principal='oryx/127.16.48.60@KRBTEST.COM'} at 127.0.0.1:42740
I20260619 16:56:10.664196 24119 tablet_replica.cc:330] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: stopping tablet replica
I20260619 16:56:10.664273 24120 raft_consensus.cc:2229] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:56:10.664426 24119 raft_consensus.cc:2229] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:10.664692 23785 tablet_service.cc:1514] Processing DeleteTablet for tablet 412d7b71e7fe44738c279b869315eef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:10 UTC) from {username='oryx', principal='oryx/127.16.48.60@KRBTEST.COM'} at 127.0.0.1:50918
I20260619 16:56:10.664772 24119 raft_consensus.cc:2258] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:10.664777 24120 raft_consensus.cc:2258] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:10.665529 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 23513
I20260619 16:56:10.666287 23925 tablet_service.cc:1514] Processing DeleteTablet for tablet 412d7b71e7fe44738c279b869315eef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:10 UTC) from {username='oryx', principal='oryx/127.16.48.60@KRBTEST.COM'} at 127.0.0.1:47588
I20260619 16:56:10.667567 24120 ts_tablet_manager.cc:1905] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:10.674146 24120 ts_tablet_manager.cc:1918] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:10.674474 24120 log.cc:1196] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/412d7b71e7fe44738c279b869315eef6
I20260619 16:56:10.674983 24120 ts_tablet_manager.cc:1939] T 412d7b71e7fe44738c279b869315eef6 P 731e6bc712914e86a3c5307622e64f51: Deleting consensus metadata
I20260619 16:56:10.675745 24121 tablet_replica.cc:330] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: stopping tablet replica
I20260619 16:56:10.676155 24121 raft_consensus.cc:2229] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:10.676548 24119 ts_tablet_manager.cc:1905] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:10.676584 24121 raft_consensus.cc:2258] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:10.678084 24121 ts_tablet_manager.cc:1905] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:56:10.681033 23509 catalog_manager.cc:4808] TS 731e6bc712914e86a3c5307622e64f51 (127.16.48.2:40595): delete failed for tablet 412d7b71e7fe44738c279b869315eef6 because the tablet was not found. No further retry: Not found: Tablet not found: 412d7b71e7fe44738c279b869315eef6
I20260619 16:56:10.683780 24121 ts_tablet_manager.cc:1918] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:10.683960 24121 log.cc:1196] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/412d7b71e7fe44738c279b869315eef6
I20260619 16:56:10.683949 24119 ts_tablet_manager.cc:1918] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:10.684207 24119 log.cc:1196] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/412d7b71e7fe44738c279b869315eef6
I20260619 16:56:10.684540 24121 ts_tablet_manager.cc:1939] T 412d7b71e7fe44738c279b869315eef6 P 7552fdd23713435ab989a0150f76e121: Deleting consensus metadata
I20260619 16:56:10.684686 24119 ts_tablet_manager.cc:1939] T 412d7b71e7fe44738c279b869315eef6 P 2d29389ddcd3472c9b6ea7e1cba2078e: Deleting consensus metadata
I20260619 16:56:10.685712 23509 catalog_manager.cc:4841] TS 7552fdd23713435ab989a0150f76e121 (127.16.48.1:39311): tablet 412d7b71e7fe44738c279b869315eef6 (table test-table [id=97785fe80a444d0db6b6ebfe89e76d1b]) successfully deleted
W20260619 16:56:10.686276 23509 catalog_manager.cc:4808] TS 2d29389ddcd3472c9b6ea7e1cba2078e (127.16.48.3:33621): delete failed for tablet 412d7b71e7fe44738c279b869315eef6 because the tablet was not found. No further retry: Not found: Tablet not found: 412d7b71e7fe44738c279b869315eef6
I20260619 16:56:10.688925 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.666472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.16.48.1:39311)
0619 16:56:10.666715 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:10.666725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:10.669702 (+ 2977us) server_negotiation.cc:407] Connection header received
0619 16:56:10.669912 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:10.669918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:10.670009 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:10.670134 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:10.672561 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.673691 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.675967 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.676736 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.676886 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.680838 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:10.680859 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:10.680881 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:10.680937 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:10.685643 (+ 4706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:10.686800 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:10.686809 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:10.686811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:10.686988 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:10.687423 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:10.687426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:10.687428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:10.687801 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:10.688110 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:10.688472 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 16:56:10.688671 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:56:10.689605 23700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:10.786924 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 23513
I20260619 16:56:10.805102 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 23722
I20260619 16:56:10.819772 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.805901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.16.48.2:40595)
0619 16:56:10.806081 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:10.806087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:10.806181 (+ 94us) server_negotiation.cc:407] Connection header received
0619 16:56:10.806403 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:10.806407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:10.806473 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:10.806587 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:10.807925 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.808747 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.810162 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.810566 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.810722 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.813481 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:10.813499 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:10.813516 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:10.813560 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:10.816732 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:10.817545 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:10.817560 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:10.817563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:10.817710 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:10.818116 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:10.818131 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:10.818139 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:10.818518 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:10.818837 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:10.819188 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 16:56:10.819417 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:56:10.820623 23845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:10.940438 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 23722
I20260619 16:56:10.962834 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 23863
I20260619 16:56:10.977702 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:10.963666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.16.48.3:33621)
0619 16:56:10.963879 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:10.963885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:10.963905 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:56:10.964011 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:10.964016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:10.964090 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:10.964248 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:10.965697 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.966642 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.968032 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:10.968554 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:10.968707 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:10.971399 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:10.971423 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:10.971446 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:10.971503 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:10.974516 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:10.975309 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:10.975323 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:10.975331 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:10.975471 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:10.975911 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:10.975917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:10.975920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:10.976399 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:10.976721 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:10.977085 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 16:56:10.977343 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 16:56:10.978524 23986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:11.075748 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 23863
I20260619 16:56:11.076637 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:11.075799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:33996)
0619 16:56:11.076289 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:11.076408 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"threads_started":1}
I20260619 16:56:11.085438 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:11.084988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:41377 (local address 127.0.0.1:34010)
0619 16:56:11.085189 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:11.085293 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:41377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 16:56:11.097548 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 23193
I20260619 16:56:11.098764 23361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:11.220743 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 23193
W20260619 16:56:11.242489 23511 connection.cc:535] client connection to 127.16.48.61:38415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:56:11.242852 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 23294
I20260619 16:56:11.243837 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:11.243377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38415 (local address 127.0.0.1:59228)
0619 16:56:11.243602 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:11.243680 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 16:56:11.243861 23556 generic_service.cc:196] Checking for leaks (request via RPC)
W20260619 16:56:11.244033 23511 consensus_peers.cc:487] T 00000000000000000000000000000000 P 693be2ed13624107816f36c365395757 -> Peer 2c609163a3874286809b0d1adc1405da (127.16.48.61:38415): Couldn't send request to peer 2c609163a3874286809b0d1adc1405da. Status: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:56:11.364074 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 23294
2026-06-19T16:56:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11591 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:56:11 dist-test-slave-3ffd krb5kdc[24132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:56:11 dist-test-slave-3ffd krb5kdc[24132](info): set up 2 sockets
Jun 19 16:56:11 dist-test-slave-3ffd krb5kdc[24132](info): commencing operation
krb5kdc: starting...
W20260619 16:56:12.478824 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:56:12 dist-test-slave-3ffd krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:56:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.62.keytab.
Entry for principal oryx/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.62.keytab.
I20260619 16:56:12.728863 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:43027
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.62.keytab
--principal=oryx/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:43027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:13.079023 24145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:13.079355 24145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:13.079533 24145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:13.087544 24145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:13.087710 24145 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:56:13.087755 24145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:13.087875 24145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:13.087975 24145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:13.088080 24145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:13.099280 24145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:43027
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.62.keytab
--principal=oryx/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:43027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:13.100930 24145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:13.102545 24145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:13.117527 24153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:13.117627 24151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:13.117656 24150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:13.118043 24145 server_base.cc:1034] running on GCE node
I20260619 16:56:13.118955 24145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:13.120748 24145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:13.122051 24145 hybrid_clock.cc:648] HybridClock initialized: now 1781888173121954 us; error 72 us; skew 500 ppm
I20260619 16:56:13.122501 24145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:13 dist-test-slave-3ffd krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888173, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:13.126598 24145 init.cc:377] Logged in from keytab as oryx/127.16.48.62@KRBTEST.COM (short username oryx)
I20260619 16:56:13.128567 24145 webserver.cc:460] Webserver started at http://127.16.48.62:36485/ using document root <none> and password file <none>
I20260619 16:56:13.129381 24145 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:13.129488 24145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:13.129809 24145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:13.133414 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "de1357d639d14c5e8904649573399a9c"
format_stamp: "Formatted at 2026-06-19 16:56:13 on dist-test-slave-3ffd"
server_key: "129d30d7b2ea32b1a41101c4bd10564b"
server_key_iv: "141b2d4048c522c9e07c6c9c77aa72ff"
server_key_version: "encryptionkey@0"
I20260619 16:56:13.134325 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "de1357d639d14c5e8904649573399a9c"
format_stamp: "Formatted at 2026-06-19 16:56:13 on dist-test-slave-3ffd"
server_key: "129d30d7b2ea32b1a41101c4bd10564b"
server_key_iv: "141b2d4048c522c9e07c6c9c77aa72ff"
server_key_version: "encryptionkey@0"
I20260619 16:56:13.140358 24145 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 16:56:13.144593 24160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:13.146433 24145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:56:13.146706 24145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "de1357d639d14c5e8904649573399a9c"
format_stamp: "Formatted at 2026-06-19 16:56:13 on dist-test-slave-3ffd"
server_key: "129d30d7b2ea32b1a41101c4bd10564b"
server_key_iv: "141b2d4048c522c9e07c6c9c77aa72ff"
server_key_version: "encryptionkey@0"
I20260619 16:56:13.147014 24145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:13.184603 24145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:13.189459 24145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:13.189810 24145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:13.214246 24145 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:43027
I20260619 16:56:13.214272 24221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:43027 every 8 connection(s)
I20260619 16:56:13.215966 24145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:56:13.220260 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:13.223939 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 24145
I20260619 16:56:13.224224 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:56:13.225036 16576 external_mini_cluster.cc:1428] Setting key 38b71afd98c0189b8e3b2bee973a7c61
I20260619 16:56:13.234182 24222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c: Bootstrap starting.
I20260619 16:56:13.237722 24222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:13.238886 24222 log.cc:824] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:56:13 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:13.242430 24222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c: No bootstrap required, opened a new log
I20260619 16:56:13.250135 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:13.227507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.16.48.62:43027)
0619 16:56:13.228427 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:13.228441 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:13.228485 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:56:13.230346 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:13.230377 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:13.230890 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:13.231399 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:13.233025 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.234480 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.235826 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.236659 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.236875 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:13.241606 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:13.241616 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:13.241633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:13.241673 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:13.245197 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:13.246247 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:13.246254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:13.246262 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:13.246475 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:13.246939 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:13.246943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:13.246945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:13.247624 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:13.247860 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:13.248449 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 16:56:13.248921 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":385,"threads_started":1}
I20260619 16:56:13.252255 24222 raft_consensus.cc:348] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1357d639d14c5e8904649573399a9c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43027 } }
I20260619 16:56:13.252697 24222 raft_consensus.cc:374] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:13.252866 24222 raft_consensus.cc:729] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1357d639d14c5e8904649573399a9c, State: Initialized, Role: FOLLOWER
I20260619 16:56:13.253546 24222 consensus_queue.cc:260] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1357d639d14c5e8904649573399a9c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43027 } }
I20260619 16:56:13.253861 24222 raft_consensus.cc:388] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:56:13.254004 24222 raft_consensus.cc:482] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:56:13.254199 24222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:13.256626 24222 raft_consensus.cc:504] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1357d639d14c5e8904649573399a9c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43027 } }
I20260619 16:56:13.257225 24222 leader_election.cc:302] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de1357d639d14c5e8904649573399a9c; no voters:
I20260619 16:56:13.257918 24222 leader_election.cc:288] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:56:13.258098 24227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:13.258898 24227 raft_consensus.cc:686] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [term 1 LEADER]: Becoming Leader. State: Replica: de1357d639d14c5e8904649573399a9c, State: Running, Role: LEADER
I20260619 16:56:13.259496 24227 consensus_queue.cc:237] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1357d639d14c5e8904649573399a9c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43027 } }
I20260619 16:56:13.260423 24222 sys_catalog.cc:564] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:56:13.263002 24229 sys_catalog.cc:455] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader de1357d639d14c5e8904649573399a9c. Latest consensus state: current_term: 1 leader_uuid: "de1357d639d14c5e8904649573399a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1357d639d14c5e8904649573399a9c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43027 } } }
I20260619 16:56:13.262897 24228 sys_catalog.cc:455] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de1357d639d14c5e8904649573399a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1357d639d14c5e8904649573399a9c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43027 } } }
I20260619 16:56:13.263330 24229 sys_catalog.cc:458] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:13.263329 24228 sys_catalog.cc:458] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:13.266211 24234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:56:13.272203 24234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:56:13.285774 24234 catalog_manager.cc:1347] Generated new cluster ID: 8af2d595e23746ecbed70f160a8a6caf
I20260619 16:56:13.285928 24234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:56:13.298985 24234 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:56:13.300503 24234 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:56:13.314324 24234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c: Generated new TSK 0
I20260619 16:56:13.315291 24234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab.
Entry for principal oryx/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab.
I20260619 16:56:13.376720 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43027
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab
--principal=oryx/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:13.757011 24248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:13.757321 24248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:13.757421 24248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:13.765254 24248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:13.765383 24248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:13.765470 24248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:56:13.765595 24248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:56:13.776324 24248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab
--principal=oryx/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.48.62:43027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:13.777690 24248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:13.779049 24248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:13.793911 24256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:13.793980 24253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:13.793949 24254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:13.794834 24248 server_base.cc:1034] running on GCE node
I20260619 16:56:13.795461 24248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:13.796566 24248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:13.797816 24248 hybrid_clock.cc:648] HybridClock initialized: now 1781888173797719 us; error 68 us; skew 500 ppm
I20260619 16:56:13.798179 24248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:13 dist-test-slave-3ffd krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888173, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:13.801856 24248 init.cc:377] Logged in from keytab as oryx/127.16.48.1@KRBTEST.COM (short username oryx)
I20260619 16:56:13.803488 24248 webserver.cc:460] Webserver started at http://127.16.48.1:41523/ using document root <none> and password file <none>
I20260619 16:56:13.804353 24248 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:13.804456 24248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:13.804755 24248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:13.807942 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "0d9f9c54a5f34434969092bbd5ba6b85"
format_stamp: "Formatted at 2026-06-19 16:56:13 on dist-test-slave-3ffd"
server_key: "500893841334a389308250727f633646"
server_key_iv: "4fa5b4366c4b2db33fe97a00de1080a5"
server_key_version: "encryptionkey@0"
I20260619 16:56:13.808914 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "0d9f9c54a5f34434969092bbd5ba6b85"
format_stamp: "Formatted at 2026-06-19 16:56:13 on dist-test-slave-3ffd"
server_key: "500893841334a389308250727f633646"
server_key_iv: "4fa5b4366c4b2db33fe97a00de1080a5"
server_key_version: "encryptionkey@0"
I20260619 16:56:13.815521 24248 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260619 16:56:13.819844 24263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:13.821674 24248 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:56:13.821936 24248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "0d9f9c54a5f34434969092bbd5ba6b85"
format_stamp: "Formatted at 2026-06-19 16:56:13 on dist-test-slave-3ffd"
server_key: "500893841334a389308250727f633646"
server_key_iv: "4fa5b4366c4b2db33fe97a00de1080a5"
server_key_version: "encryptionkey@0"
I20260619 16:56:13.822214 24248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:13.858096 24248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:13.863351 24248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:13.863689 24248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:13.866874 24248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:13.867009 24248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:13.867175 24248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:13.867241 24248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:56:13 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888173, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:13.892915 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:13.867922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:47079 (local address 127.16.48.62:43027)
0619 16:56:13.868439 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:13.868443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:13.869893 (+ 1450us) server_negotiation.cc:407] Connection header received
0619 16:56:13.872288 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:13.872298 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:13.872437 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:13.872576 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:13.875230 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.876337 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.878920 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.879352 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.879515 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:13.885297 (+ 5782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:13.885310 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:13.885317 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:13.885378 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:13.889219 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:13.890131 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:13.890139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:13.890143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:13.890287 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:13.890813 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:13.890818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:13.890822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:13.891111 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:13.891281 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:13.892164 (+ 883us) server_negotiation.cc:299] Negotiation successful
0619 16:56:13.892531 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":300,"threads_started":1}
I20260619 16:56:13.894933 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:13.868711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43027 (local address 127.16.48.1:47079)
0619 16:56:13.869612 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:13.869755 (+ 143us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:13.871721 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:13.872920 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:13.872939 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:13.873786 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:13.874979 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:13.875018 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.876527 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.876538 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:13.878724 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:13.878743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.879623 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.879633 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:13.879910 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:13.880805 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:13.880851 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:13.884864 (+ 4013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:13.889429 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:13.889440 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:13.889467 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:13.889969 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:13.890421 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:13.890427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:13.890431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:13.890673 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:13.891259 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:13.891268 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:13.891710 (+ 442us) client_negotiation.cc:769] Sending connection context
0619 16:56:13.892019 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 16:56:13.892705 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":411,"threads_started":1}
I20260619 16:56:13.918937 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:13.902923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43027 (local address 127.16.48.1:34187)
0619 16:56:13.903765 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:13.903793 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:13.904007 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:13.904525 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:13.904533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:13.905348 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:13.905861 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:13.905881 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.907172 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.907178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:13.909059 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:13.909087 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.910224 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.910231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:13.910416 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:13.911143 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:13.911161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:13.911970 (+ 809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:13.915341 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:13.915350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:13.915355 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:13.915984 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:13.916563 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:13.916572 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:13.916576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:13.916814 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:13.917610 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:13.917622 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:13.917829 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 16:56:13.918032 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 16:56:13.918474 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":436,"threads_started":1}
I20260619 16:56:13.919508 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:13.903246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:34187 (local address 127.16.48.62:43027)
0619 16:56:13.903466 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:13.903473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:13.903927 (+ 454us) server_negotiation.cc:407] Connection header received
0619 16:56:13.904217 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:13.904226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:13.904329 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:13.904454 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:13.906107 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.906946 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.909314 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:13.910001 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:13.910179 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:13.912251 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:13.912259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:13.912267 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:13.912318 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:13.915158 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:13.916215 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:13.916222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:13.916225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:13.916422 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:13.916993 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:13.916998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:13.917001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:13.917345 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:13.917524 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:13.918767 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0619 16:56:13.919089 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 16:56:13.925231 24248 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:46111
I20260619 16:56:13.925248 24383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:46111 every 8 connection(s)
I20260619 16:56:13.926546 24248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:56:13.928800 24384 heartbeater.cc:347] Connected to a master server at 127.16.48.62:43027
I20260619 16:56:13.929014 24384 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:13.929682 24384 heartbeater.cc:511] Master 127.16.48.62:43027 requested a full tablet report, sending...
I20260619 16:56:13.931870 24177 ts_manager.cc:194] Registered new tserver with Master: 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111)
I20260619 16:56:13.933601 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 24248
I20260619 16:56:13.933821 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:56:13.933959 24177 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:47079
I20260619 16:56:13.934656 16576 external_mini_cluster.cc:1428] Setting key 7a22b9ae391e89a31aa87a5855491c6c
WARNING: no policy specified for oryx/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab.
Entry for principal oryx/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab.
I20260619 16:56:13.988251 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43027
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab
--principal=oryx/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:14.349701 24388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:14.350014 24388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:14.350112 24388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:14.358129 24388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:14.358274 24388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:14.358366 24388 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:56:14.358440 24388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:56:14.369184 24388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab
--principal=oryx/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.48.62:43027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:14.370575 24388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:14.372053 24388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:14.386838 24396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:14.386816 24394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:14.387125 24393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:14.387876 24388 server_base.cc:1034] running on GCE node
I20260619 16:56:14.388690 24388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:14.389796 24388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:14.391109 24388 hybrid_clock.cc:648] HybridClock initialized: now 1781888174391055 us; error 41 us; skew 500 ppm
I20260619 16:56:14.391479 24388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:14 dist-test-slave-3ffd krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888174, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:14.395041 24388 init.cc:377] Logged in from keytab as oryx/127.16.48.2@KRBTEST.COM (short username oryx)
I20260619 16:56:14.396924 24388 webserver.cc:460] Webserver started at http://127.16.48.2:46383/ using document root <none> and password file <none>
I20260619 16:56:14.397689 24388 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:14.397782 24388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:14.398064 24388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:14.401209 24388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "ef241f2a813846d9820141be016f56fc"
format_stamp: "Formatted at 2026-06-19 16:56:14 on dist-test-slave-3ffd"
server_key: "43d90f3e096d9867960e768625ccef52"
server_key_iv: "7bbcdf67fb0ede1ba79c4aa94db61dfa"
server_key_version: "encryptionkey@0"
I20260619 16:56:14.402007 24388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "ef241f2a813846d9820141be016f56fc"
format_stamp: "Formatted at 2026-06-19 16:56:14 on dist-test-slave-3ffd"
server_key: "43d90f3e096d9867960e768625ccef52"
server_key_iv: "7bbcdf67fb0ede1ba79c4aa94db61dfa"
server_key_version: "encryptionkey@0"
I20260619 16:56:14.407774 24388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:56:14.412261 24403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:14.413933 24388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:14.414151 24388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "ef241f2a813846d9820141be016f56fc"
format_stamp: "Formatted at 2026-06-19 16:56:14 on dist-test-slave-3ffd"
server_key: "43d90f3e096d9867960e768625ccef52"
server_key_iv: "7bbcdf67fb0ede1ba79c4aa94db61dfa"
server_key_version: "encryptionkey@0"
I20260619 16:56:14.414419 24388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:14.450986 24388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:14.456346 24388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:14.456678 24388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:14.459759 24388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:14.459892 24388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:14.460075 24388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:14.460208 24388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 16:56:14 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888174, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:14.487459 24412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:14.460457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:58785 (local address 127.16.48.62:43027)
0619 16:56:14.464012 (+ 3555us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:14.464017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:14.464039 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:56:14.466931 (+ 2892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:14.466940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:14.467057 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:14.467235 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:14.469548 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:14.470631 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:14.473234 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:14.473927 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:14.474096 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:14.479548 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:14.479554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:14.479557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:14.479597 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:14.483040 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:14.483959 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:14.483967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:14.483973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:14.484200 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:14.484877 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:14.484885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:14.484891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:14.485391 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:14.485606 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:14.486484 (+ 878us) server_negotiation.cc:299] Negotiation successful
0619 16:56:14.486852 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3444,"thread_start_us":3347,"threads_started":1}
I20260619 16:56:14.488185 24413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:14.461685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43027 (local address 127.16.48.2:58785)
0619 16:56:14.462863 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:14.462930 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:14.466438 (+ 3508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:14.467536 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:14.467546 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:14.468232 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:14.469213 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:14.469264 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:14.470854 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:14.470860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:14.473014 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:14.473033 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:14.474233 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:14.474242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:14.474514 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:14.475418 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:14.475446 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:14.479221 (+ 3775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:14.483241 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:14.483253 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:14.483277 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:14.483786 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:14.484357 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:14.484362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:14.484364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:14.484679 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:14.485562 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:14.485569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:14.485993 (+ 424us) client_negotiation.cc:769] Sending connection context
0619 16:56:14.486271 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 16:56:14.486739 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":488,"threads_started":1}
I20260619 16:56:14.515254 24523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:41957 every 8 connection(s)
I20260619 16:56:14.515285 24388 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:41957
I20260619 16:56:14.517176 24388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:56:14.519013 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:14.499707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43027 (local address 127.16.48.2:39883)
0619 16:56:14.500489 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:14.500515 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:14.500723 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:14.501238 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:14.501243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:14.501716 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:14.502021 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:14.502031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:14.503736 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:14.503743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:14.506173 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:14.506194 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:14.507792 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:14.507817 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:14.508086 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:14.509086 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:14.509122 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:14.510712 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:14.515566 (+ 4854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:14.515573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:14.515578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:14.516190 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:14.516665 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:14.516673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:14.516679 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:14.516852 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:14.517574 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:14.517579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:14.517859 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 16:56:14.518044 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 16:56:14.518499 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":443,"threads_started":1}
I20260619 16:56:14.519887 24524 heartbeater.cc:347] Connected to a master server at 127.16.48.62:43027
I20260619 16:56:14.520025 24412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:14.499745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:39883 (local address 127.16.48.62:43027)
0619 16:56:14.499998 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:14.500008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:14.500621 (+ 613us) server_negotiation.cc:407] Connection header received
0619 16:56:14.500913 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:14.500920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:14.501037 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:14.501186 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:14.502217 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:14.503511 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:14.506433 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:14.507274 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:14.507470 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:14.511024 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:14.511033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:14.511042 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:14.511103 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:14.515380 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:14.516369 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:14.516374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:14.516376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:14.516523 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:14.517001 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:14.517005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:14.517007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:14.517447 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:14.517568 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:14.519431 (+ 1863us) server_negotiation.cc:299] Negotiation successful
0619 16:56:14.519732 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:56:14.520172 24524 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:14.521006 24524 heartbeater.cc:511] Master 127.16.48.62:43027 requested a full tablet report, sending...
I20260619 16:56:14.522455 24177 ts_manager.cc:194] Registered new tserver with Master: ef241f2a813846d9820141be016f56fc (127.16.48.2:41957)
I20260619 16:56:14.523587 24177 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:58785
I20260619 16:56:14.524739 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 24388
I20260619 16:56:14.524926 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:56:14.525390 16576 external_mini_cluster.cc:1428] Setting key 69f325142347b24dbc245cac0fe6c578
WARNING: no policy specified for oryx/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab.
Entry for principal oryx/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab.
I20260619 16:56:14.577644 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43027
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab
--principal=oryx/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:14.935039 24528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:14.935355 24528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:14.935554 24528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:56:14.938094 24384 heartbeater.cc:503] Master 127.16.48.62:43027 was elected leader, sending a full tablet report...
W20260619 16:56:14.943843 24528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:14.943976 24528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:14.944132 24528 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:56:14.944303 24528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:56:14.954797 24528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab
--principal=oryx/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.48.62:43027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:14.956219 24528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:14.957669 24528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:14.972335 24536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:14.972470 24533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:14.972537 24534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:14.973526 24528 server_base.cc:1034] running on GCE node
I20260619 16:56:14.974238 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:14.975369 24528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:14.976679 24528 hybrid_clock.cc:648] HybridClock initialized: now 1781888174976578 us; error 75 us; skew 500 ppm
I20260619 16:56:14.977041 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:14 dist-test-slave-3ffd krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888174, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:14.980556 24528 init.cc:377] Logged in from keytab as oryx/127.16.48.3@KRBTEST.COM (short username oryx)
I20260619 16:56:14.982321 24528 webserver.cc:460] Webserver started at http://127.16.48.3:34375/ using document root <none> and password file <none>
I20260619 16:56:14.983107 24528 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:14.983248 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:14.983630 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:14.986869 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "83781e3e6e384aba96a9209cf1d408ce"
format_stamp: "Formatted at 2026-06-19 16:56:14 on dist-test-slave-3ffd"
server_key: "bd46285522bf8bf1d35582acd3950552"
server_key_iv: "225abe3cc9d08630fc6af661fdf9fdd2"
server_key_version: "encryptionkey@0"
I20260619 16:56:14.987764 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "83781e3e6e384aba96a9209cf1d408ce"
format_stamp: "Formatted at 2026-06-19 16:56:14 on dist-test-slave-3ffd"
server_key: "bd46285522bf8bf1d35582acd3950552"
server_key_iv: "225abe3cc9d08630fc6af661fdf9fdd2"
server_key_version: "encryptionkey@0"
I20260619 16:56:14.994513 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:56:14.999444 24543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:15.001344 24528 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:56:15.001610 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "83781e3e6e384aba96a9209cf1d408ce"
format_stamp: "Formatted at 2026-06-19 16:56:14 on dist-test-slave-3ffd"
server_key: "bd46285522bf8bf1d35582acd3950552"
server_key_iv: "225abe3cc9d08630fc6af661fdf9fdd2"
server_key_version: "encryptionkey@0"
I20260619 16:56:15.001890 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:15.018178 24528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:15.023267 24528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:15.023597 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:15.026840 24528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:15.026964 24528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:15.027150 24528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:15.027230 24528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:56:15 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888174, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:15.056829 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.027557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:50489 (local address 127.16.48.62:43027)
0619 16:56:15.028525 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.028532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.029683 (+ 1151us) server_negotiation.cc:407] Connection header received
0619 16:56:15.032034 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.032040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.032169 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.032300 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:15.035555 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.036772 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.040495 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.041347 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.041558 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.046913 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:15.046923 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:15.046930 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:15.046988 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:15.051539 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:15.053247 (+ 1708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:15.053253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:15.053258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:15.053462 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:15.054114 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:15.054120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:15.054123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:15.054574 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:15.054755 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.056097 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.056416 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":699,"threads_started":1}
I20260619 16:56:15.058557 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.028482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43027 (local address 127.16.48.3:50489)
0619 16:56:15.029420 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.029494 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.031668 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.032710 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.032726 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.033899 (+ 1173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:15.035267 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.035310 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.037117 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.037127 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.040187 (+ 3060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.040222 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.041690 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.041701 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.041961 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.042891 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:15.042932 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:15.046522 (+ 3590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:15.051788 (+ 5266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:15.051810 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:15.051844 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:15.052971 (+ 1127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:15.053661 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:15.053669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:15.053675 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:15.053941 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:15.054763 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:15.054775 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:15.055366 (+ 591us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.055803 (+ 437us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.056626 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":417,"threads_started":1}
I20260619 16:56:15.073568 24528 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:35121
I20260619 16:56:15.073642 24663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:35121 every 8 connection(s)
I20260619 16:56:15.075382 24528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:56:15.078261 24664 heartbeater.cc:347] Connected to a master server at 127.16.48.62:43027
I20260619 16:56:15.078531 24664 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:15.079298 24664 heartbeater.cc:511] Master 127.16.48.62:43027 requested a full tablet report, sending...
I20260619 16:56:15.080971 24177 ts_manager.cc:194] Registered new tserver with Master: 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121)
I20260619 16:56:15.081840 24177 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:50489
I20260619 16:56:15.082870 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 24528
I20260619 16:56:15.083066 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:56:15.083618 16576 external_mini_cluster.cc:1428] Setting key 976c027f0895a1dbf97fa886f9bf2f78
I20260619 16:56:15.083752 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.067378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:49695 (local address 127.16.48.62:43027)
0619 16:56:15.067995 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.068005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.068523 (+ 518us) server_negotiation.cc:407] Connection header received
0619 16:56:15.068706 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.068714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.068812 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.068952 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:15.070602 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.071777 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.074152 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.074797 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.074965 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.077352 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:15.077359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:15.077364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:15.077407 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:15.080734 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:15.081518 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:15.081524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:15.081528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:15.081675 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:15.082116 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:15.082121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:15.082124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:15.082445 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:15.082626 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.083148 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.083394 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260619 16:56:15.083810 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.067443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43027 (local address 127.16.48.3:49695)
0619 16:56:15.068394 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.068414 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.068564 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.069035 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.069042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.069837 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:15.070373 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.070395 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.072017 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.072026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.073966 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.073983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.075015 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.075021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.075166 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.075848 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:15.075867 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:15.077103 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:15.080900 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:15.080906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:15.080910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:15.081375 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:15.081809 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:15.081814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:15.081817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:15.081952 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:15.082591 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:15.082596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:15.082799 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.082978 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.083343 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":557,"threads_started":1}
I20260619 16:56:15.087246 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:56:15 dist-test-slave-3ffd krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:56:15 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:15 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:56:15.142100 24552 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:15.131082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.16.48.62:43027)
0619 16:56:15.131353 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.131359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.131511 (+ 152us) server_negotiation.cc:407] Connection header received
0619 16:56:15.131719 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.131724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.131811 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.131910 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:15.133333 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.134103 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.135526 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.135972 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.136087 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.141863 (+ 5776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36464: BlockingRecv error: recv got EOF from 127.0.0.1:36464 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Jun 19 16:56:15 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:15.160617 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.145558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.16.48.62:43027)
0619 16:56:15.145738 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.145743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.146105 (+ 362us) server_negotiation.cc:407] Connection header received
0619 16:56:15.146323 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.146327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.146397 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.146496 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:15.147800 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.148702 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.150062 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.150547 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.150661 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.154823 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:15.154829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:15.154833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:15.154875 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:15.157576 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:15.158551 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:15.158560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:15.158565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:15.158756 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:15.159229 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:15.159236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:15.159239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:15.159573 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:15.159803 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.160089 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.160382 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:56:15.165218 24177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:56:15.172900 24177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:56:15.200525 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.187110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.16.48.1:46111)
0619 16:56:15.188543 (+ 1433us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.188553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.189291 (+ 738us) server_negotiation.cc:407] Connection header received
0619 16:56:15.189979 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.189986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.190260 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.190407 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:15.191795 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.193405 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.197218 (+ 3813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.198733 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.198854 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.199033 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.199981 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.200200 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":1128,"threads_started":1}
I20260619 16:56:15.200562 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.187092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.16.48.3:35121)
0619 16:56:15.188066 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.188081 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.189292 (+ 1211us) server_negotiation.cc:407] Connection header received
0619 16:56:15.189765 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.189775 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.190086 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.190288 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:15.191513 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.193405 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.196930 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.198607 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.198759 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.199042 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.200000 (+ 958us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.200225 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":466,"threads_started":1}
I20260619 16:56:15.200606 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.186577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:35121 (local address 127.0.0.1:36012)
0619 16:56:15.189126 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.189176 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.189455 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.190752 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.190761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.190794 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:15.191262 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.191277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.193657 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.193663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.196691 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.196705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.199228 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.199241 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.199583 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.199739 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.199891 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.200141 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2234,"mutex_wait_us":131,"thread_start_us":2190,"threads_started":1}
I20260619 16:56:15.201476 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.186915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:46111 (local address 127.0.0.1:34190)
0619 16:56:15.189120 (+ 2205us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.189176 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.189803 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.191103 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.191108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.191131 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:15.191591 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.191607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.193657 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.193663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.197012 (+ 3349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.197027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.199259 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.199264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.199583 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.199739 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.200847 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.200996 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1960,"mutex_wait_us":112,"spinlock_wait_cycles":844032,"thread_start_us":3133,"threads_started":1}
I20260619 16:56:15.203233 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.189407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41957 (local address 127.0.0.1:44142)
0619 16:56:15.194399 (+ 4992us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.194420 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.194548 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.195996 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.196001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.196025 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:15.196559 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.196575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.198574 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.198579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.200619 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.200637 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.202426 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.202433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.202629 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.202672 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.202751 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.202890 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4851,"thread_start_us":4666,"threads_started":1}
I20260619 16:56:15.204839 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.189817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.16.48.2:41957)
0619 16:56:15.195087 (+ 5270us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.195096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.195131 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:56:15.195236 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.195248 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.195557 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.195785 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:15.196806 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.198378 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.200836 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.202185 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.204035 (+ 1850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.204295 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.204375 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.204547 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5085,"thread_start_us":4940,"threads_started":1}
I20260619 16:56:15.205339 24314 tablet_service.cc:1467] Processing CreateTablet for tablet e8b8d011fe5f432e925e032a2b78bf23 (DEFAULT_TABLE table=test-table [id=08ccd6beb0494ed1b897533b9a5655fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:15.205344 24594 tablet_service.cc:1467] Processing CreateTablet for tablet e8b8d011fe5f432e925e032a2b78bf23 (DEFAULT_TABLE table=test-table [id=08ccd6beb0494ed1b897533b9a5655fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:15.206727 24594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b8d011fe5f432e925e032a2b78bf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:15.206759 24314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b8d011fe5f432e925e032a2b78bf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:15.208801 24454 tablet_service.cc:1467] Processing CreateTablet for tablet e8b8d011fe5f432e925e032a2b78bf23 (DEFAULT_TABLE table=test-table [id=08ccd6beb0494ed1b897533b9a5655fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:15.210271 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b8d011fe5f432e925e032a2b78bf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:15.226982 24688 tablet_bootstrap.cc:492] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: Bootstrap starting.
I20260619 16:56:15.230890 24688 tablet_bootstrap.cc:654] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:15.230842 24689 tablet_bootstrap.cc:492] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: Bootstrap starting.
I20260619 16:56:15.232300 24688 log.cc:824] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:15.235918 24689 tablet_bootstrap.cc:654] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:15.237210 24688 tablet_bootstrap.cc:492] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: No bootstrap required, opened a new log
I20260619 16:56:15.237541 24688 ts_tablet_manager.cc:1397] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 16:56:15.237798 24689 log.cc:824] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:15.239154 24690 tablet_bootstrap.cc:492] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: Bootstrap starting.
I20260619 16:56:15.242805 24690 tablet_bootstrap.cc:654] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:15.244257 24690 log.cc:824] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:15.245971 24689 tablet_bootstrap.cc:492] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: No bootstrap required, opened a new log
I20260619 16:56:15.246279 24689 ts_tablet_manager.cc:1397] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20260619 16:56:15.246892 24690 tablet_bootstrap.cc:492] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: No bootstrap required, opened a new log
I20260619 16:56:15.247170 24690 ts_tablet_manager.cc:1397] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:56:15.248219 24688 raft_consensus.cc:348] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } }
I20260619 16:56:15.248704 24688 raft_consensus.cc:374] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:15.248802 24688 raft_consensus.cc:729] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83781e3e6e384aba96a9209cf1d408ce, State: Initialized, Role: FOLLOWER
I20260619 16:56:15.249478 24688 consensus_queue.cc:260] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } }
I20260619 16:56:15.250716 24664 heartbeater.cc:503] Master 127.16.48.62:43027 was elected leader, sending a full tablet report...
I20260619 16:56:15.251463 24688 ts_tablet_manager.cc:1428] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 16:56:15.255940 24689 raft_consensus.cc:348] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } }
I20260619 16:56:15.256474 24689 raft_consensus.cc:374] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:15.256583 24689 raft_consensus.cc:729] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9f9c54a5f34434969092bbd5ba6b85, State: Initialized, Role: FOLLOWER
I20260619 16:56:15.257282 24689 consensus_queue.cc:260] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } }
I20260619 16:56:15.257160 24690 raft_consensus.cc:348] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } }
I20260619 16:56:15.257633 24690 raft_consensus.cc:374] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:15.257750 24690 raft_consensus.cc:729] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef241f2a813846d9820141be016f56fc, State: Initialized, Role: FOLLOWER
I20260619 16:56:15.258420 24690 consensus_queue.cc:260] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } }
I20260619 16:56:15.259529 24689 ts_tablet_manager.cc:1428] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 16:56:15.260181 24524 heartbeater.cc:503] Master 127.16.48.62:43027 was elected leader, sending a full tablet report...
I20260619 16:56:15.260206 24690 ts_tablet_manager.cc:1428] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: Time spent starting tablet: real 0.013s user 0.006s sys 0.009s
W20260619 16:56:15.269620 24525 tablet.cc:2369] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:56:15.328037 24665 tablet.cc:2369] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:56:15.429701 24385 tablet.cc:2369] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:56:15.451216 24695 raft_consensus.cc:482] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:56:15.451625 24695 raft_consensus.cc:504] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } }
I20260619 16:56:15.453441 24695 leader_election.cc:288] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121), ef241f2a813846d9820141be016f56fc (127.16.48.2:41957)
I20260619 16:56:15.460760 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.453985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:43441 (local address 127.16.48.2:41957)
0619 16:56:15.454205 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.454211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.455022 (+ 811us) server_negotiation.cc:407] Connection header received
0619 16:56:15.455228 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.455232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.455329 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.455426 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:15.456215 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.456965 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.458793 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.459566 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.459746 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.459869 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.460321 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.460470 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:56:15.460810 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.453829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:53041 (local address 127.16.48.3:35121)
0619 16:56:15.454052 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.454062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.454865 (+ 803us) server_negotiation.cc:407] Connection header received
0619 16:56:15.455153 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.455161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.455245 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.455317 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:15.455995 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.456799 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.458794 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.459566 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.459761 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.459887 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.460355 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.460491 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:56:15.460944 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.453797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:35121 (local address 127.16.48.1:53041)
0619 16:56:15.454703 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.454723 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.454938 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.455381 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.455387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.455417 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:15.455834 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.455846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.456949 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.456954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.458619 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.458632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.459839 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.459848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.459997 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.460079 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.460293 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.460478 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":89,"thread_start_us":486,"threads_started":1}
I20260619 16:56:15.460969 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.454035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41957 (local address 127.16.48.1:43441)
0619 16:56:15.454901 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.454916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.455077 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.455482 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.455486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.455518 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:15.455981 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.455993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.457117 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.457120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.458619 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.458632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.459887 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.459895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.460046 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.460069 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.460262 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.460477 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":72,"thread_start_us":543,"threads_started":1}
I20260619 16:56:15.462073 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b8d011fe5f432e925e032a2b78bf23" candidate_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83781e3e6e384aba96a9209cf1d408ce" is_pre_election: true
I20260619 16:56:15.462267 24477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b8d011fe5f432e925e032a2b78bf23" candidate_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef241f2a813846d9820141be016f56fc" is_pre_election: true
I20260619 16:56:15.462736 24614 raft_consensus.cc:2454] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9f9c54a5f34434969092bbd5ba6b85 in term 0.
I20260619 16:56:15.462769 24477 raft_consensus.cc:2454] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9f9c54a5f34434969092bbd5ba6b85 in term 0.
I20260619 16:56:15.463464 24265 leader_election.cc:302] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9f9c54a5f34434969092bbd5ba6b85, 83781e3e6e384aba96a9209cf1d408ce; no voters:
I20260619 16:56:15.463932 24695 raft_consensus.cc:2781] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:56:15.464061 24695 raft_consensus.cc:482] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:56:15.464208 24695 raft_consensus.cc:3037] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:15.466694 24695 raft_consensus.cc:504] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } }
I20260619 16:56:15.467365 24695 leader_election.cc:288] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [CANDIDATE]: Term 1 election: Requested vote from peers 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121), ef241f2a813846d9820141be016f56fc (127.16.48.2:41957)
I20260619 16:56:15.468024 24477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b8d011fe5f432e925e032a2b78bf23" candidate_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef241f2a813846d9820141be016f56fc"
I20260619 16:56:15.468042 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b8d011fe5f432e925e032a2b78bf23" candidate_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83781e3e6e384aba96a9209cf1d408ce"
I20260619 16:56:15.468323 24477 raft_consensus.cc:3037] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:15.468348 24614 raft_consensus.cc:3037] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:15.470599 24614 raft_consensus.cc:2454] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9f9c54a5f34434969092bbd5ba6b85 in term 1.
I20260619 16:56:15.470598 24477 raft_consensus.cc:2454] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9f9c54a5f34434969092bbd5ba6b85 in term 1.
I20260619 16:56:15.471189 24265 leader_election.cc:302] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d9f9c54a5f34434969092bbd5ba6b85, 83781e3e6e384aba96a9209cf1d408ce, ef241f2a813846d9820141be016f56fc; no voters:
I20260619 16:56:15.471596 24695 raft_consensus.cc:2781] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:15.472082 24695 raft_consensus.cc:686] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 LEADER]: Becoming Leader. State: Replica: 0d9f9c54a5f34434969092bbd5ba6b85, State: Running, Role: LEADER
I20260619 16:56:15.472653 24695 consensus_queue.cc:237] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } }
I20260619 16:56:15.477240 24176 catalog_manager.cc:5495] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1). New cstate: current_term: 1 leader_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } health_report { overall_health: UNKNOWN } } }
I20260619 16:56:15.513190 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.507553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.16.48.1:46111)
0619 16:56:15.507766 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.507771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.507788 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:56:15.508041 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.508045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.508222 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.508361 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:15.508694 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.509370 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.510832 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.511236 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.511331 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.511897 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:15.512506 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:15.512622 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.512787 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.512856 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:56:15.522840 24614 raft_consensus.cc:1261] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Refusing update from remote peer 0d9f9c54a5f34434969092bbd5ba6b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:15.522840 24477 raft_consensus.cc:1261] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Refusing update from remote peer 0d9f9c54a5f34434969092bbd5ba6b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:15.524015 24699 consensus_queue.cc:1035] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:15.524458 24695 consensus_queue.cc:1035] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:15.538220 24702 mvcc.cc:205] Tried to move back new op lower bound from 7298613966930784256 to 7298613966739755008. Current Snapshot: MvccSnapshot[applied={T|T < 7298613966930784256}]
I20260619 16:56:15.543488 24704 mvcc.cc:205] Tried to move back new op lower bound from 7298613966930784256 to 7298613966739755008. Current Snapshot: MvccSnapshot[applied={T|T < 7298613966930784256}]
I20260619 16:56:15.554874 24177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:15.555197 24177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:15.559122 24177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c: Sending DeleteTablet for 3 replicas of tablet e8b8d011fe5f432e925e032a2b78bf23
I20260619 16:56:15.560410 24594 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b8d011fe5f432e925e032a2b78bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:15 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:36012
I20260619 16:56:15.561110 24454 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b8d011fe5f432e925e032a2b78bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:15 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:44142
I20260619 16:56:15.561203 24314 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b8d011fe5f432e925e032a2b78bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:15 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:34190
I20260619 16:56:15.561619 24713 tablet_replica.cc:330] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: stopping tablet replica
I20260619 16:56:15.562163 24713 raft_consensus.cc:2229] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:15.562805 24713 raft_consensus.cc:2258] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:15.562961 24176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:56:15.563711 24176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:56:15.565635 24713 ts_tablet_manager.cc:1905] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:15.573104 24713 ts_tablet_manager.cc:1918] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:15.573467 24713 log.cc:1196] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/e8b8d011fe5f432e925e032a2b78bf23
I20260619 16:56:15.573401 24594 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdc3ffa8288472a833fbe51bf3c7734 (DEFAULT_TABLE table=test-table [id=bab9a044b14b43a19a689e99d449e476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:15.574128 24594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdc3ffa8288472a833fbe51bf3c7734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:15.574337 24713 ts_tablet_manager.cc:1939] T e8b8d011fe5f432e925e032a2b78bf23 P 83781e3e6e384aba96a9209cf1d408ce: Deleting consensus metadata
I20260619 16:56:15.576866 24162 catalog_manager.cc:4841] TS 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121): tablet e8b8d011fe5f432e925e032a2b78bf23 (table test-table [id=08ccd6beb0494ed1b897533b9a5655fd]) successfully deleted
I20260619 16:56:15.577096 24313 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdc3ffa8288472a833fbe51bf3c7734 (DEFAULT_TABLE table=test-table [id=bab9a044b14b43a19a689e99d449e476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:15.577798 24313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdc3ffa8288472a833fbe51bf3c7734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:15.578091 24453 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdc3ffa8288472a833fbe51bf3c7734 (DEFAULT_TABLE table=test-table [id=bab9a044b14b43a19a689e99d449e476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:15.578653 24453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdc3ffa8288472a833fbe51bf3c7734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:15.585721 24689 tablet_bootstrap.cc:492] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: Bootstrap starting.
I20260619 16:56:15.588275 24689 tablet_bootstrap.cc:654] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:15.589470 24688 tablet_bootstrap.cc:492] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: Bootstrap starting.
I20260619 16:56:15.591135 24688 tablet_bootstrap.cc:654] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:15.601579 24688 tablet_bootstrap.cc:492] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: No bootstrap required, opened a new log
I20260619 16:56:15.601892 24688 ts_tablet_manager.cc:1397] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260619 16:56:15.603088 24690 tablet_bootstrap.cc:492] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: Bootstrap starting.
I20260619 16:56:15.605314 24690 tablet_bootstrap.cc:654] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:15.602811 24688 raft_consensus.cc:348] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.608146 24688 raft_consensus.cc:374] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:15.608551 24688 raft_consensus.cc:729] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83781e3e6e384aba96a9209cf1d408ce, State: Initialized, Role: FOLLOWER
I20260619 16:56:15.608985 24715 tablet_replica.cc:330] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: stopping tablet replica
I20260619 16:56:15.609378 24688 consensus_queue.cc:260] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.609548 24715 raft_consensus.cc:2229] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:56:15.611186 24688 ts_tablet_manager.cc:1428] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260619 16:56:15.613422 24715 raft_consensus.cc:2258] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:15.615294 24714 tablet_replica.cc:330] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: stopping tablet replica
I20260619 16:56:15.615924 24714 raft_consensus.cc:2229] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:15.617795 24714 raft_consensus.cc:2258] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:15.618446 24690 tablet_bootstrap.cc:492] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: No bootstrap required, opened a new log
I20260619 16:56:15.618659 24690 ts_tablet_manager.cc:1397] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20260619 16:56:15.619608 24690 raft_consensus.cc:348] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.619946 24690 raft_consensus.cc:374] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:15.620150 24690 raft_consensus.cc:729] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef241f2a813846d9820141be016f56fc, State: Initialized, Role: FOLLOWER
I20260619 16:56:15.620512 24715 ts_tablet_manager.cc:1905] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:15.620488 24690 consensus_queue.cc:260] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.620877 24689 tablet_bootstrap.cc:492] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: No bootstrap required, opened a new log
I20260619 16:56:15.621308 24689 ts_tablet_manager.cc:1397] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.019s
I20260619 16:56:15.622077 24690 ts_tablet_manager.cc:1428] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:56:15.626045 24689 raft_consensus.cc:348] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.626349 24689 raft_consensus.cc:374] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:15.626463 24689 raft_consensus.cc:729] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9f9c54a5f34434969092bbd5ba6b85, State: Initialized, Role: FOLLOWER
I20260619 16:56:15.627408 24714 ts_tablet_manager.cc:1905] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:15.628324 24715 ts_tablet_manager.cc:1918] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:15.628552 24715 log.cc:1196] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/e8b8d011fe5f432e925e032a2b78bf23
I20260619 16:56:15.628357 24689 consensus_queue.cc:260] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.629470 24689 ts_tablet_manager.cc:1428] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260619 16:56:15.629961 24715 ts_tablet_manager.cc:1939] T e8b8d011fe5f432e925e032a2b78bf23 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting consensus metadata
I20260619 16:56:15.631531 24163 catalog_manager.cc:4841] TS 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111): tablet e8b8d011fe5f432e925e032a2b78bf23 (table test-table [id=08ccd6beb0494ed1b897533b9a5655fd]) successfully deleted
I20260619 16:56:15.636451 24714 ts_tablet_manager.cc:1918] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:15.636654 24714 log.cc:1196] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/e8b8d011fe5f432e925e032a2b78bf23
I20260619 16:56:15.637239 24714 ts_tablet_manager.cc:1939] T e8b8d011fe5f432e925e032a2b78bf23 P ef241f2a813846d9820141be016f56fc: Deleting consensus metadata
I20260619 16:56:15.639235 24164 catalog_manager.cc:4841] TS ef241f2a813846d9820141be016f56fc (127.16.48.2:41957): tablet e8b8d011fe5f432e925e032a2b78bf23 (table test-table [id=08ccd6beb0494ed1b897533b9a5655fd]) successfully deleted
I20260619 16:56:15.712360 24694 raft_consensus.cc:482] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:56:15.712654 24694 raft_consensus.cc:504] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.714274 24694 leader_election.cc:288] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111), ef241f2a813846d9820141be016f56fc (127.16.48.2:41957)
I20260619 16:56:15.722561 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.714815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:33025 (local address 127.16.48.2:41957)
0619 16:56:15.715049 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.715058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.715715 (+ 657us) server_negotiation.cc:407] Connection header received
0619 16:56:15.716064 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.716070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.716207 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.716292 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:15.717135 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.717992 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.720453 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.721490 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.721628 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.721733 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.722105 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.722252 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:56:15.722657 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.714519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41957 (local address 127.16.48.3:33025)
0619 16:56:15.715554 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.715580 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.715842 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.716467 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.716474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.716518 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:15.716913 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.716931 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.718184 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.718190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.720261 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.720278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.721681 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.721688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.721893 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.721932 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.722052 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.722211 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":58,"spinlock_wait_cycles":31488,"thread_start_us":715,"threads_started":1}
I20260619 16:56:15.723755 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.714523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:46111 (local address 127.16.48.3:38715)
0619 16:56:15.715842 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.715865 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.715967 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.716514 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.716517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.716534 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:15.716935 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.716944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.718510 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.718516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.720800 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.720818 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.722724 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.722730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.723120 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.723155 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.723245 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.723372 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"mutex_wait_us":490,"thread_start_us":503,"threads_started":1}
I20260619 16:56:15.723767 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.714781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:38715 (local address 127.16.48.1:46111)
0619 16:56:15.714979 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.714985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.716080 (+ 1095us) server_negotiation.cc:407] Connection header received
0619 16:56:15.716192 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.716199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.716287 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.716370 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:15.717108 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.718315 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.721029 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.722266 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.722428 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.722525 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.723310 (+ 785us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.723464 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:56:15.723927 24477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdc3ffa8288472a833fbe51bf3c7734" candidate_uuid: "83781e3e6e384aba96a9209cf1d408ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef241f2a813846d9820141be016f56fc" is_pre_election: true
I20260619 16:56:15.724243 24477 raft_consensus.cc:2454] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83781e3e6e384aba96a9209cf1d408ce in term 0.
I20260619 16:56:15.724785 24547 leader_election.cc:302] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83781e3e6e384aba96a9209cf1d408ce, ef241f2a813846d9820141be016f56fc; no voters:
I20260619 16:56:15.724768 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdc3ffa8288472a833fbe51bf3c7734" candidate_uuid: "83781e3e6e384aba96a9209cf1d408ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" is_pre_election: true
I20260619 16:56:15.725239 24694 raft_consensus.cc:2781] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:56:15.725363 24694 raft_consensus.cc:482] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:56:15.725462 24694 raft_consensus.cc:3037] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:15.725416 24334 raft_consensus.cc:2454] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83781e3e6e384aba96a9209cf1d408ce in term 0.
I20260619 16:56:15.728844 24694 raft_consensus.cc:504] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.729650 24694 leader_election.cc:288] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [CANDIDATE]: Term 1 election: Requested vote from peers 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111), ef241f2a813846d9820141be016f56fc (127.16.48.2:41957)
I20260619 16:56:15.730273 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdc3ffa8288472a833fbe51bf3c7734" candidate_uuid: "83781e3e6e384aba96a9209cf1d408ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9f9c54a5f34434969092bbd5ba6b85"
I20260619 16:56:15.730298 24477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdc3ffa8288472a833fbe51bf3c7734" candidate_uuid: "83781e3e6e384aba96a9209cf1d408ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef241f2a813846d9820141be016f56fc"
I20260619 16:56:15.730516 24477 raft_consensus.cc:3037] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:15.730536 24334 raft_consensus.cc:3037] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:15.733047 24334 raft_consensus.cc:2454] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83781e3e6e384aba96a9209cf1d408ce in term 1.
I20260619 16:56:15.733608 24477 raft_consensus.cc:2454] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83781e3e6e384aba96a9209cf1d408ce in term 1.
I20260619 16:56:15.733614 24546 leader_election.cc:302] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9f9c54a5f34434969092bbd5ba6b85, 83781e3e6e384aba96a9209cf1d408ce; no voters:
I20260619 16:56:15.734011 24694 raft_consensus.cc:2781] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:15.734557 24694 raft_consensus.cc:686] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 1 LEADER]: Becoming Leader. State: Replica: 83781e3e6e384aba96a9209cf1d408ce, State: Running, Role: LEADER
I20260619 16:56:15.734968 24694 consensus_queue.cc:237] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.738845 24176 catalog_manager.cc:5495] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3). New cstate: current_term: 1 leader_uuid: "83781e3e6e384aba96a9209cf1d408ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } health_report { overall_health: HEALTHY } } }
Jun 19 16:56:15 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888173, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.62@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:15.800766 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.778884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.16.48.62:43027)
0619 16:56:15.779583 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.779594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.779834 (+ 240us) server_negotiation.cc:407] Connection header received
0619 16:56:15.780062 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.780067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.780192 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.780318 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:15.781437 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.782208 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.783708 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.784183 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.784325 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.794955 (+ 10630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:15.794960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:15.794963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:15.794997 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:15.797839 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:15.798652 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:15.798658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:15.798662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:15.798783 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:15.799161 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:15.799166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:15.799170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:15.799461 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:15.799616 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.800029 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.800417 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":188,"server-negotiator.queue_time_us":506,"thread_start_us":338,"threads_started":1}
I20260619 16:56:15.800765 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.778884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43027 (local address 127.0.0.1:36476)
0619 16:56:15.779721 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.779737 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.779919 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.780339 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.780343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.781073 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:15.781285 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.781295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.782370 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.782376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.783562 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.783575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.784367 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.784382 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.784590 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.792078 (+ 7488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:15.792162 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:15.794700 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:15.798009 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:15.798026 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:15.798034 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:15.798475 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:15.798905 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:15.798909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:15.798913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:15.799041 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:15.799604 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:15.799612 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:15.799785 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.799957 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.800305 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":2,"thread_start_us":557,"threads_started":1}
I20260619 16:56:15.807147 24176 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:36476:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:56:15.808418 24176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:56:15.816275 24454 tablet_service.cc:1467] Processing CreateTablet for tablet 7f204107b44844d196f0546ff878034d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc19cc59898d44c7b000c4040e0eb643]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:56:15.816324 24314 tablet_service.cc:1467] Processing CreateTablet for tablet 7f204107b44844d196f0546ff878034d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc19cc59898d44c7b000c4040e0eb643]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:56:15.816452 24594 tablet_service.cc:1467] Processing CreateTablet for tablet 7f204107b44844d196f0546ff878034d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc19cc59898d44c7b000c4040e0eb643]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:56:15.816900 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f204107b44844d196f0546ff878034d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:15.816936 24594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f204107b44844d196f0546ff878034d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:15.817082 24314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f204107b44844d196f0546ff878034d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:15.824478 24690 tablet_bootstrap.cc:492] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: Bootstrap starting.
I20260619 16:56:15.825456 24689 tablet_bootstrap.cc:492] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: Bootstrap starting.
I20260619 16:56:15.826383 24690 tablet_bootstrap.cc:654] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:15.827405 24689 tablet_bootstrap.cc:654] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:15.829074 24688 tablet_bootstrap.cc:492] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: Bootstrap starting.
I20260619 16:56:15.830657 24690 tablet_bootstrap.cc:492] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: No bootstrap required, opened a new log
I20260619 16:56:15.830725 24688 tablet_bootstrap.cc:654] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:15.830850 24690 ts_tablet_manager.cc:1397] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260619 16:56:15.831835 24690 raft_consensus.cc:348] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.832209 24690 raft_consensus.cc:374] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:15.832315 24690 raft_consensus.cc:729] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef241f2a813846d9820141be016f56fc, State: Initialized, Role: FOLLOWER
I20260619 16:56:15.832722 24690 consensus_queue.cc:260] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.833323 24696 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.833873 24696 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:56:15.834417 24689 tablet_bootstrap.cc:492] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: No bootstrap required, opened a new log
I20260619 16:56:15.834633 24689 ts_tablet_manager.cc:1397] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 16:56:15.835443 24689 raft_consensus.cc:348] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.835744 24689 raft_consensus.cc:374] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:15.835835 24689 raft_consensus.cc:729] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9f9c54a5f34434969092bbd5ba6b85, State: Initialized, Role: FOLLOWER
I20260619 16:56:15.837496 24689 consensus_queue.cc:260] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.837569 24696 raft_consensus.cc:482] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:56:15.837821 24696 raft_consensus.cc:504] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.838181 24689 ts_tablet_manager.cc:1428] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260619 16:56:15.839578 24695 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.839879 24695 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:56:15.839964 24696 leader_election.cc:288] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111), 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121)
I20260619 16:56:15.840458 24690 ts_tablet_manager.cc:1428] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260619 16:56:15.840590 24688 tablet_bootstrap.cc:492] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: No bootstrap required, opened a new log
I20260619 16:56:15.840782 24688 ts_tablet_manager.cc:1397] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260619 16:56:15.846577 24688 raft_consensus.cc:348] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.846896 24688 raft_consensus.cc:374] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:15.846977 24688 raft_consensus.cc:729] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83781e3e6e384aba96a9209cf1d408ce, State: Initialized, Role: FOLLOWER
I20260619 16:56:15.847280 24688 consensus_queue.cc:260] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.847716 24694 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.848052 24694 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:56:15.852260 24688 ts_tablet_manager.cc:1428] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20260619 16:56:15.861356 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.842811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:50851 (local address 127.16.48.3:35121)
0619 16:56:15.843031 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.843038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.854355 (+ 11317us) server_negotiation.cc:407] Connection header received
0619 16:56:15.854695 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.854702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.854806 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.854872 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:15.855556 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.856485 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.858837 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.860083 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.860259 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.860383 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.860838 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.860987 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 16:56:15.861768 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.840777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:56075 (local address 127.16.48.1:46111)
0619 16:56:15.840944 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.840950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.851420 (+ 10470us) server_negotiation.cc:407] Connection header received
0619 16:56:15.851546 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.851556 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.851716 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.851831 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:15.855026 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.856283 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.859169 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.860641 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.860820 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.860928 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.861320 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.861506 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:56:15.862005 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.840433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:46111 (local address 127.16.48.2:56075)
0619 16:56:15.851133 (+ 10700us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.851156 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.851311 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.854273 (+ 2962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.854279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.854319 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:15.854750 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.854763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.856494 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.856500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.858955 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.858977 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.860890 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.860896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.861122 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.861157 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.861288 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.861490 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10554,"spinlock_wait_cycles":2944,"thread_start_us":10401,"threads_started":1}
I20260619 16:56:15.862381 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.841789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:35121 (local address 127.16.48.2:50851)
0619 16:56:15.854252 (+ 12463us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.854272 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.854504 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.855064 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.855067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.855090 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:15.855360 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.855366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.856667 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.856674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.858626 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.858644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.860402 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.860409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.860624 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.860663 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.861681 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.861834 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12332,"mutex_wait_us":5,"spinlock_wait_cycles":3584,"thread_start_us":12138,"threads_started":1}
I20260619 16:56:15.862869 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f204107b44844d196f0546ff878034d" candidate_uuid: "ef241f2a813846d9820141be016f56fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" is_pre_election: true
I20260619 16:56:15.863179 24334 raft_consensus.cc:2454] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef241f2a813846d9820141be016f56fc in term 0.
I20260619 16:56:15.863298 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f204107b44844d196f0546ff878034d" candidate_uuid: "ef241f2a813846d9820141be016f56fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83781e3e6e384aba96a9209cf1d408ce" is_pre_election: true
I20260619 16:56:15.863565 24614 raft_consensus.cc:2454] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef241f2a813846d9820141be016f56fc in term 0.
I20260619 16:56:15.863730 24406 leader_election.cc:302] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9f9c54a5f34434969092bbd5ba6b85, ef241f2a813846d9820141be016f56fc; no voters:
I20260619 16:56:15.864245 24696 raft_consensus.cc:2781] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:56:15.864424 24696 raft_consensus.cc:482] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:56:15.864750 24696 raft_consensus.cc:3037] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:15.867424 24696 raft_consensus.cc:504] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.868149 24696 leader_election.cc:288] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [CANDIDATE]: Term 1 election: Requested vote from peers 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111), 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121)
I20260619 16:56:15.868739 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f204107b44844d196f0546ff878034d" candidate_uuid: "ef241f2a813846d9820141be016f56fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83781e3e6e384aba96a9209cf1d408ce"
I20260619 16:56:15.868968 24614 raft_consensus.cc:3037] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:15.871071 24614 raft_consensus.cc:2454] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef241f2a813846d9820141be016f56fc in term 1.
I20260619 16:56:15.871569 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f204107b44844d196f0546ff878034d" candidate_uuid: "ef241f2a813846d9820141be016f56fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9f9c54a5f34434969092bbd5ba6b85"
I20260619 16:56:15.871627 24405 leader_election.cc:302] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83781e3e6e384aba96a9209cf1d408ce, ef241f2a813846d9820141be016f56fc; no voters:
I20260619 16:56:15.871867 24334 raft_consensus.cc:3037] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:15.871989 24696 raft_consensus.cc:2781] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:15.872534 24696 raft_consensus.cc:686] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [term 1 LEADER]: Becoming Leader. State: Replica: ef241f2a813846d9820141be016f56fc, State: Running, Role: LEADER
I20260619 16:56:15.873274 24696 consensus_queue.cc:237] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:15.875221 24334 raft_consensus.cc:2454] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef241f2a813846d9820141be016f56fc in term 1.
I20260619 16:56:15.877509 24177 catalog_manager.cc:5495] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc reported cstate change: term changed from 0 to 1, leader changed from <none> to ef241f2a813846d9820141be016f56fc (127.16.48.2). New cstate: current_term: 1 leader_uuid: "ef241f2a813846d9820141be016f56fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } health_report { overall_health: UNKNOWN } } }
I20260619 16:56:15.877784 24735 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: TxnStatusTablet state changed. Reason: New leader ef241f2a813846d9820141be016f56fc. Latest consensus state: current_term: 1 leader_uuid: "ef241f2a813846d9820141be016f56fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.878145 24735 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:56:15.878902 24737 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:56:15.879164 24737 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:56:15.908788 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.900348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41957 (local address 127.0.0.1:44146)
0619 16:56:15.900763 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.900785 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.900960 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.901591 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.901596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.901611 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:56:15.901845 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.901858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.903477 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.903485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.905270 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.905289 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.906400 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.906407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.906677 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.906772 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:56:15.908039 (+ 1267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:56:15.908070 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.908205 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.908432 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260619 16:56:15.909534 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.900342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.16.48.2:41957)
0619 16:56:15.900535 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.900542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.900890 (+ 348us) server_negotiation.cc:407] Connection header received
0619 16:56:15.901231 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.901240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.901361 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.901543 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:15.902029 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.903286 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.905479 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.906166 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.906334 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.907024 (+ 690us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:15.907900 (+ 876us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:15.908080 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.909132 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.909248 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:56:15.914235 24614 raft_consensus.cc:1261] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Refusing update from remote peer ef241f2a813846d9820141be016f56fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:15.914592 24334 raft_consensus.cc:1261] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Refusing update from remote peer ef241f2a813846d9820141be016f56fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:15.915860 24696 consensus_queue.cc:1035] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:15.916863 24740 consensus_queue.cc:1035] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:15.930222 24695 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: TxnStatusTablet state changed. Reason: New leader ef241f2a813846d9820141be016f56fc. Latest consensus state: current_term: 1 leader_uuid: "ef241f2a813846d9820141be016f56fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.930526 24695 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:56:15.931218 24703 mvcc.cc:205] Tried to move back new op lower bound from 7298613968536694784 to 7298613968381394944. Current Snapshot: MvccSnapshot[applied={T|T < 7298613968536694784}]
I20260619 16:56:15.932551 24696 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef241f2a813846d9820141be016f56fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.932880 24696 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:56:15.937871 24694 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: TxnStatusTablet state changed. Reason: New leader ef241f2a813846d9820141be016f56fc. Latest consensus state: current_term: 1 leader_uuid: "ef241f2a813846d9820141be016f56fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.938230 24694 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:56:15.943553 24695 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef241f2a813846d9820141be016f56fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.943863 24695 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P 0d9f9c54a5f34434969092bbd5ba6b85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:56:15.945959 24743 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef241f2a813846d9820141be016f56fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.946358 24743 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P ef241f2a813846d9820141be016f56fc: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:56:15.946974 24694 tablet_replica.cc:439] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef241f2a813846d9820141be016f56fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } } }
I20260619 16:56:15.947285 24694 tablet_replica.cc:442] T 7f204107b44844d196f0546ff878034d P 83781e3e6e384aba96a9209cf1d408ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:56:15.949716 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.938543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.16.48.3:35121)
0619 16:56:15.938731 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.938737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.938896 (+ 159us) server_negotiation.cc:407] Connection header received
0619 16:56:15.939156 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.939161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.939278 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.939452 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:15.940082 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.941280 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.943258 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.943898 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.944039 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.944923 (+ 884us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:15.948826 (+ 3903us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:15.949016 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.949316 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.949420 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:56:15.990659 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.982807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:32871 (local address 127.16.48.2:41957)
0619 16:56:15.983023 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:15.983031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:15.983253 (+ 222us) server_negotiation.cc:407] Connection header received
0619 16:56:15.983629 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:15.983635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:15.983730 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:15.983844 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:15.984599 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.985839 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.988055 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.988882 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.989037 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.989709 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:15.989858 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:15.989957 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:15.990173 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 16:56:15.990310 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:56:15.991272 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:15.982276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41957 (local address 127.16.48.3:32871)
0619 16:56:15.983113 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:15.983137 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:15.983408 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:15.983913 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:15.983919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:15.983934 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:56:15.984395 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.984412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.986041 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.986049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.987826 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:15.987846 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:15.989143 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:15.989151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:15.989371 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:15.989464 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:56:15.990001 (+ 537us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:56:15.990014 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 16:56:15.990116 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:56:15.990294 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":755,"threads_started":1}
I20260619 16:56:16.005723 24334 raft_consensus.cc:1261] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Refusing update from remote peer 83781e3e6e384aba96a9209cf1d408ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:16.006739 24721 consensus_queue.cc:1035] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:16.009600 24477 raft_consensus.cc:1261] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Refusing update from remote peer 83781e3e6e384aba96a9209cf1d408ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:16.010406 24721 consensus_queue.cc:1035] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:16.069643 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:16.059743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:37745 (local address 127.16.48.3:35121)
0619 16:56:16.059927 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:16.059934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:16.062298 (+ 2364us) server_negotiation.cc:407] Connection header received
0619 16:56:16.062541 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:16.062546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:16.062639 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:16.062706 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:16.064431 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.065310 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.067202 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.067890 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.068017 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.068756 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:16.068909 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:16.068978 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:16.069200 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 16:56:16.069295 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:56:16.070626 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:16.059175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:35121 (local address 127.16.48.2:37745)
0619 16:56:16.062163 (+ 2988us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:16.062190 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:16.062385 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:16.063213 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:16.063217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:16.063230 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:56:16.063830 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:16.064241 (+ 411us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.065787 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.065791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:16.067018 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:16.067032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.068211 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.068217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:16.068416 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.068505 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:56:16.069059 (+ 554us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:56:16.069068 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:56:16.070142 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0619 16:56:16.070282 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2682,"thread_start_us":2493,"threads_started":1}
I20260619 16:56:16.172559 24175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:16.172888 24175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:16.177246 24175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c: Sending DeleteTablet for 3 replicas of tablet 0fdc3ffa8288472a833fbe51bf3c7734
I20260619 16:56:16.178506 24314 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fdc3ffa8288472a833fbe51bf3c7734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:16 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:34190
I20260619 16:56:16.179553 24594 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fdc3ffa8288472a833fbe51bf3c7734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:16 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:36012
I20260619 16:56:16.179757 24755 tablet_replica.cc:330] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: stopping tablet replica
I20260619 16:56:16.179952 24755 raft_consensus.cc:2229] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:16.180246 24755 raft_consensus.cc:2258] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:16.181005 24454 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fdc3ffa8288472a833fbe51bf3c7734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:16 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:44142
I20260619 16:56:16.195457 24757 tablet_replica.cc:330] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: stopping tablet replica
I20260619 16:56:16.196236 24755 ts_tablet_manager.cc:1905] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:16.196561 24757 raft_consensus.cc:2229] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:16.196938 24757 raft_consensus.cc:2258] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:16.205217 24756 tablet_replica.cc:330] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: stopping tablet replica
I20260619 16:56:16.205628 24756 raft_consensus.cc:2229] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:56:16.206352 24756 raft_consensus.cc:2258] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:16.206370 24757 ts_tablet_manager.cc:1905] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:16.207736 24756 ts_tablet_manager.cc:1905] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:16.208513 24755 ts_tablet_manager.cc:1918] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:56:16.208698 24755 log.cc:1196] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/0fdc3ffa8288472a833fbe51bf3c7734
I20260619 16:56:16.209178 24755 ts_tablet_manager.cc:1939] T 0fdc3ffa8288472a833fbe51bf3c7734 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting consensus metadata
I20260619 16:56:16.214793 24757 ts_tablet_manager.cc:1918] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:56:16.214960 24757 log.cc:1196] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/0fdc3ffa8288472a833fbe51bf3c7734
I20260619 16:56:16.215512 24757 ts_tablet_manager.cc:1939] T 0fdc3ffa8288472a833fbe51bf3c7734 P ef241f2a813846d9820141be016f56fc: Deleting consensus metadata
I20260619 16:56:16.215704 24756 ts_tablet_manager.cc:1918] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:56:16.215839 24756 log.cc:1196] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/0fdc3ffa8288472a833fbe51bf3c7734
I20260619 16:56:16.216288 24756 ts_tablet_manager.cc:1939] T 0fdc3ffa8288472a833fbe51bf3c7734 P 83781e3e6e384aba96a9209cf1d408ce: Deleting consensus metadata
I20260619 16:56:16.217325 24162 catalog_manager.cc:4841] TS 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121): tablet 0fdc3ffa8288472a833fbe51bf3c7734 (table test-table [id=bab9a044b14b43a19a689e99d449e476]) successfully deleted
I20260619 16:56:16.217325 24163 catalog_manager.cc:4841] TS 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111): tablet 0fdc3ffa8288472a833fbe51bf3c7734 (table test-table [id=bab9a044b14b43a19a689e99d449e476]) successfully deleted
I20260619 16:56:16.220980 24164 catalog_manager.cc:4841] TS ef241f2a813846d9820141be016f56fc (127.16.48.2:41957): tablet 0fdc3ffa8288472a833fbe51bf3c7734 (table test-table [id=bab9a044b14b43a19a689e99d449e476]) successfully deleted
Jun 19 16:56:16 dist-test-slave-3ffd krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:56:16 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:16 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:56:16.280155 24728 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:16.266286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.16.48.62:43027)
0619 16:56:16.266486 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:16.266493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:16.267055 (+ 562us) server_negotiation.cc:407] Connection header received
0619 16:56:16.267208 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:16.267214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:16.267308 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:16.267433 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:16.269353 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.270503 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.272539 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.273227 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.273397 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.279826 (+ 6429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36484: BlockingRecv error: recv got EOF from 127.0.0.1:36484 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:56:16 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM
I20260619 16:56:16.307477 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:16.285848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.16.48.62:43027)
0619 16:56:16.286029 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:16.286037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:16.286555 (+ 518us) server_negotiation.cc:407] Connection header received
0619 16:56:16.286748 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:16.286756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:16.286846 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:16.286959 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:16.288627 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.289784 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.291796 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.292509 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.292662 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.298249 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:16.298257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:16.298263 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:16.298336 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:16.302350 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:16.303429 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:16.303436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:16.303440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:16.303669 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:16.304288 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:16.304294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:16.304299 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:16.304683 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:16.304973 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:16.306797 (+ 1824us) server_negotiation.cc:299] Negotiation successful
0619 16:56:16.307085 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:56:16.312220 24175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:56:16.313010 24175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:56:16.322347 24314 tablet_service.cc:1467] Processing CreateTablet for tablet c3faee7973c941fd981913f5488f0227 (DEFAULT_TABLE table=test-table [id=b0a62d74ea40495b882160b7e377cb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:16.322548 24454 tablet_service.cc:1467] Processing CreateTablet for tablet c3faee7973c941fd981913f5488f0227 (DEFAULT_TABLE table=test-table [id=b0a62d74ea40495b882160b7e377cb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:16.322866 24314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3faee7973c941fd981913f5488f0227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:16.323006 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3faee7973c941fd981913f5488f0227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:16.325290 24594 tablet_service.cc:1467] Processing CreateTablet for tablet c3faee7973c941fd981913f5488f0227 (DEFAULT_TABLE table=test-table [id=b0a62d74ea40495b882160b7e377cb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:16.325753 24594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3faee7973c941fd981913f5488f0227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:16.330660 24690 tablet_bootstrap.cc:492] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: Bootstrap starting.
I20260619 16:56:16.332536 24690 tablet_bootstrap.cc:654] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:16.342563 24688 tablet_bootstrap.cc:492] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: Bootstrap starting.
I20260619 16:56:16.342689 24690 tablet_bootstrap.cc:492] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: No bootstrap required, opened a new log
I20260619 16:56:16.342854 24690 ts_tablet_manager.cc:1397] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260619 16:56:16.343734 24690 raft_consensus.cc:348] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.344030 24690 raft_consensus.cc:374] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:16.344162 24690 raft_consensus.cc:729] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef241f2a813846d9820141be016f56fc, State: Initialized, Role: FOLLOWER
I20260619 16:56:16.344623 24688 tablet_bootstrap.cc:654] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:16.344472 24690 consensus_queue.cc:260] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.348520 24690 ts_tablet_manager.cc:1428] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260619 16:56:16.348609 24771 tablet_bootstrap.cc:492] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Bootstrap starting.
I20260619 16:56:16.350414 24771 tablet_bootstrap.cc:654] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:16.356395 24688 tablet_bootstrap.cc:492] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: No bootstrap required, opened a new log
I20260619 16:56:16.356688 24688 ts_tablet_manager.cc:1397] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260619 16:56:16.357703 24688 raft_consensus.cc:348] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.358126 24688 raft_consensus.cc:374] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:16.358285 24688 raft_consensus.cc:729] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83781e3e6e384aba96a9209cf1d408ce, State: Initialized, Role: FOLLOWER
I20260619 16:56:16.358656 24688 consensus_queue.cc:260] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.359408 24688 ts_tablet_manager.cc:1428] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:56:16.363579 24771 tablet_bootstrap.cc:492] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: No bootstrap required, opened a new log
I20260619 16:56:16.363801 24771 ts_tablet_manager.cc:1397] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260619 16:56:16.364748 24771 raft_consensus.cc:348] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.365111 24771 raft_consensus.cc:374] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:16.365302 24771 raft_consensus.cc:729] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9f9c54a5f34434969092bbd5ba6b85, State: Initialized, Role: FOLLOWER
I20260619 16:56:16.365572 24771 consensus_queue.cc:260] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.366252 24771 ts_tablet_manager.cc:1428] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:56:16.490181 24696 raft_consensus.cc:482] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:56:16.490430 24696 raft_consensus.cc:504] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.491257 24696 leader_election.cc:288] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111), 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121)
I20260619 16:56:16.491927 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3faee7973c941fd981913f5488f0227" candidate_uuid: "ef241f2a813846d9820141be016f56fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" is_pre_election: true
I20260619 16:56:16.491935 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3faee7973c941fd981913f5488f0227" candidate_uuid: "ef241f2a813846d9820141be016f56fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83781e3e6e384aba96a9209cf1d408ce" is_pre_election: true
I20260619 16:56:16.492223 24334 raft_consensus.cc:2454] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef241f2a813846d9820141be016f56fc in term 0.
I20260619 16:56:16.492290 24614 raft_consensus.cc:2454] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef241f2a813846d9820141be016f56fc in term 0.
I20260619 16:56:16.492779 24406 leader_election.cc:302] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9f9c54a5f34434969092bbd5ba6b85, ef241f2a813846d9820141be016f56fc; no voters:
I20260619 16:56:16.493099 24696 raft_consensus.cc:2781] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:56:16.493214 24696 raft_consensus.cc:482] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:56:16.493340 24696 raft_consensus.cc:3037] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:16.495710 24696 raft_consensus.cc:504] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.496457 24696 leader_election.cc:288] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [CANDIDATE]: Term 1 election: Requested vote from peers 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111), 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121)
I20260619 16:56:16.496951 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3faee7973c941fd981913f5488f0227" candidate_uuid: "ef241f2a813846d9820141be016f56fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9f9c54a5f34434969092bbd5ba6b85"
I20260619 16:56:16.497017 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3faee7973c941fd981913f5488f0227" candidate_uuid: "ef241f2a813846d9820141be016f56fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83781e3e6e384aba96a9209cf1d408ce"
I20260619 16:56:16.497171 24334 raft_consensus.cc:3037] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:16.497269 24614 raft_consensus.cc:3037] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:16.499341 24334 raft_consensus.cc:2454] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef241f2a813846d9820141be016f56fc in term 1.
I20260619 16:56:16.499851 24406 leader_election.cc:302] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9f9c54a5f34434969092bbd5ba6b85, ef241f2a813846d9820141be016f56fc; no voters:
I20260619 16:56:16.500180 24696 raft_consensus.cc:2781] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:16.500437 24696 raft_consensus.cc:686] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 1 LEADER]: Becoming Leader. State: Replica: ef241f2a813846d9820141be016f56fc, State: Running, Role: LEADER
I20260619 16:56:16.500447 24614 raft_consensus.cc:2454] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef241f2a813846d9820141be016f56fc in term 1.
I20260619 16:56:16.500721 24696 consensus_queue.cc:237] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.503670 24176 catalog_manager.cc:5495] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc reported cstate change: term changed from 0 to 1, leader changed from <none> to ef241f2a813846d9820141be016f56fc (127.16.48.2). New cstate: current_term: 1 leader_uuid: "ef241f2a813846d9820141be016f56fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } health_report { overall_health: UNKNOWN } } }
I20260619 16:56:16.550285 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:16.542094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.16.48.2:41957)
0619 16:56:16.544390 (+ 2296us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:16.544395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:16.544414 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:56:16.544479 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:16.544483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:16.544569 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:16.544677 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:16.545245 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.546012 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.548250 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.548706 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.548824 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.549445 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:16.549570 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:16.549653 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:16.549866 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 16:56:16.549943 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2186,"thread_start_us":2082,"threads_started":1}
I20260619 16:56:16.554171 24614 raft_consensus.cc:1261] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Refusing update from remote peer ef241f2a813846d9820141be016f56fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:16.554172 24334 raft_consensus.cc:1261] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Refusing update from remote peer ef241f2a813846d9820141be016f56fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:16.554841 24696 consensus_queue.cc:1035] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:16.555168 24740 consensus_queue.cc:1035] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:16.575734 24175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:16.576049 24175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:16.580230 24175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c: Sending DeleteTablet for 3 replicas of tablet c3faee7973c941fd981913f5488f0227
I20260619 16:56:16.581363 24314 tablet_service.cc:1514] Processing DeleteTablet for tablet c3faee7973c941fd981913f5488f0227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:16 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:34190
I20260619 16:56:16.581714 24755 tablet_replica.cc:330] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: stopping tablet replica
I20260619 16:56:16.581938 24755 raft_consensus.cc:2229] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:16.582113 24755 pending_rounds.cc:62] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Trying to abort 2 pending ops.
I20260619 16:56:16.582407 24594 tablet_service.cc:1514] Processing DeleteTablet for tablet c3faee7973c941fd981913f5488f0227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:16 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:36012
I20260619 16:56:16.582250 24755 pending_rounds.cc:68] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298613971159265280 op_type: WRITE_OP write_request { tablet_id: "c3faee7973c941fd981913f5488f0227" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\334\353\325\321\006\"8\n\ntest-admin\022*\n b0a62d74ea40495b882160b7e377cb39\020\001\030\001 \001(\001" signature: "<redacted>""%e\365\034\324 \253\233\t\0263]&e\363\343om\027\010\030\263K\\\355\307\\k]?[\367\014\323\336\301\003S\217\306\"\375e\303\022<\004Z\373\344\256dqX>6} \232J\265%}\272" signing_key_seq_num: 0 } } request_id { client_id: "d485b24a0bcb45a3b7d2aee5b6dced46" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260619 16:56:16.582733 24756 tablet_replica.cc:330] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: stopping tablet replica
I20260619 16:56:16.582957 24756 raft_consensus.cc:2229] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:16.582985 24755 pending_rounds.cc:68] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298613970949693440 op_type: NO_OP noop_request { }
I20260619 16:56:16.583146 24755 raft_consensus.cc:2866] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260619 16:56:16.583222 24756 raft_consensus.cc:2258] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:16.583292 24755 raft_consensus.cc:2258] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:16.584872 24755 ts_tablet_manager.cc:1905] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:16.585582 24756 ts_tablet_manager.cc:1905] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:16.589757 24454 tablet_service.cc:1514] Processing DeleteTablet for tablet c3faee7973c941fd981913f5488f0227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:16 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:44142
I20260619 16:56:16.590101 24757 tablet_replica.cc:330] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: stopping tablet replica
I20260619 16:56:16.590323 24757 raft_consensus.cc:2229] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:56:16.590904 24757 raft_consensus.cc:2258] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:16.592377 24757 ts_tablet_manager.cc:1905] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:16.598646 24176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:56:16.599504 24176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:56:16.599691 24755 ts_tablet_manager.cc:1918] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:16.599850 24755 log.cc:1196] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/c3faee7973c941fd981913f5488f0227
I20260619 16:56:16.600323 24755 ts_tablet_manager.cc:1939] T c3faee7973c941fd981913f5488f0227 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting consensus metadata
I20260619 16:56:16.601009 24163 catalog_manager.cc:4841] TS 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111): tablet c3faee7973c941fd981913f5488f0227 (table test-table [id=b0a62d74ea40495b882160b7e377cb39]) successfully deleted
I20260619 16:56:16.605336 24757 ts_tablet_manager.cc:1918] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:16.605556 24757 log.cc:1196] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/c3faee7973c941fd981913f5488f0227
I20260619 16:56:16.606027 24757 ts_tablet_manager.cc:1939] T c3faee7973c941fd981913f5488f0227 P ef241f2a813846d9820141be016f56fc: Deleting consensus metadata
I20260619 16:56:16.608834 24314 tablet_service.cc:1467] Processing CreateTablet for tablet 45c0968817954965b905696546f111f0 (DEFAULT_TABLE table=test-table [id=f4278dc69ef34581bad21d3329710d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:16.609341 24314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c0968817954965b905696546f111f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:16.612739 24164 catalog_manager.cc:4841] TS ef241f2a813846d9820141be016f56fc (127.16.48.2:41957): tablet c3faee7973c941fd981913f5488f0227 (table test-table [id=b0a62d74ea40495b882160b7e377cb39]) successfully deleted
I20260619 16:56:16.613164 24594 tablet_service.cc:1467] Processing CreateTablet for tablet 45c0968817954965b905696546f111f0 (DEFAULT_TABLE table=test-table [id=f4278dc69ef34581bad21d3329710d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:16.613250 24454 tablet_service.cc:1467] Processing CreateTablet for tablet 45c0968817954965b905696546f111f0 (DEFAULT_TABLE table=test-table [id=f4278dc69ef34581bad21d3329710d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:16.613685 24594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c0968817954965b905696546f111f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:16.613731 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c0968817954965b905696546f111f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:16.622745 24688 tablet_bootstrap.cc:492] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: Bootstrap starting.
I20260619 16:56:16.624892 24688 tablet_bootstrap.cc:654] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:16.625335 24690 tablet_bootstrap.cc:492] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: Bootstrap starting.
I20260619 16:56:16.627418 24690 tablet_bootstrap.cc:654] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:16.632097 24756 ts_tablet_manager.cc:1918] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:16.632632 24771 tablet_bootstrap.cc:492] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: Bootstrap starting.
I20260619 16:56:16.634465 24771 tablet_bootstrap.cc:654] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:16.637073 24771 tablet_bootstrap.cc:492] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: No bootstrap required, opened a new log
I20260619 16:56:16.637259 24771 ts_tablet_manager.cc:1397] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:56:16.638080 24771 raft_consensus.cc:348] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.638371 24771 raft_consensus.cc:374] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:16.638458 24771 raft_consensus.cc:729] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9f9c54a5f34434969092bbd5ba6b85, State: Initialized, Role: FOLLOWER
I20260619 16:56:16.638751 24771 consensus_queue.cc:260] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.639452 24771 ts_tablet_manager.cc:1428] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 16:56:16.640183 24756 log.cc:1196] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/c3faee7973c941fd981913f5488f0227
I20260619 16:56:16.640697 24756 ts_tablet_manager.cc:1939] T c3faee7973c941fd981913f5488f0227 P 83781e3e6e384aba96a9209cf1d408ce: Deleting consensus metadata
I20260619 16:56:16.641562 24688 tablet_bootstrap.cc:492] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: No bootstrap required, opened a new log
I20260619 16:56:16.641763 24688 ts_tablet_manager.cc:1397] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260619 16:56:16.642592 24688 raft_consensus.cc:348] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.642891 24688 raft_consensus.cc:374] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:16.642968 24688 raft_consensus.cc:729] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83781e3e6e384aba96a9209cf1d408ce, State: Initialized, Role: FOLLOWER
I20260619 16:56:16.643761 24162 catalog_manager.cc:4841] TS 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121): tablet c3faee7973c941fd981913f5488f0227 (table test-table [id=b0a62d74ea40495b882160b7e377cb39]) successfully deleted
I20260619 16:56:16.644202 24688 consensus_queue.cc:260] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.646741 24688 ts_tablet_manager.cc:1428] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:56:16.649499 24690 tablet_bootstrap.cc:492] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: No bootstrap required, opened a new log
I20260619 16:56:16.649739 24690 ts_tablet_manager.cc:1397] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.005s
I20260619 16:56:16.650513 24690 raft_consensus.cc:348] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.650849 24690 raft_consensus.cc:374] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:16.652632 24690 raft_consensus.cc:729] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef241f2a813846d9820141be016f56fc, State: Initialized, Role: FOLLOWER
I20260619 16:56:16.653028 24690 consensus_queue.cc:260] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.653611 24690 ts_tablet_manager.cc:1428] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 16:56:16.842583 24695 raft_consensus.cc:482] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:56:16.842835 24695 raft_consensus.cc:504] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.843570 24695 leader_election.cc:288] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef241f2a813846d9820141be016f56fc (127.16.48.2:41957), 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121)
I20260619 16:56:16.844265 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c0968817954965b905696546f111f0" candidate_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83781e3e6e384aba96a9209cf1d408ce" is_pre_election: true
I20260619 16:56:16.844242 24476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c0968817954965b905696546f111f0" candidate_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef241f2a813846d9820141be016f56fc" is_pre_election: true
I20260619 16:56:16.844535 24614 raft_consensus.cc:2454] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9f9c54a5f34434969092bbd5ba6b85 in term 0.
I20260619 16:56:16.844549 24476 raft_consensus.cc:2454] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9f9c54a5f34434969092bbd5ba6b85 in term 0.
I20260619 16:56:16.845095 24265 leader_election.cc:302] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9f9c54a5f34434969092bbd5ba6b85, 83781e3e6e384aba96a9209cf1d408ce; no voters:
I20260619 16:56:16.845474 24695 raft_consensus.cc:2781] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:56:16.845589 24695 raft_consensus.cc:482] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:56:16.845669 24695 raft_consensus.cc:3037] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:16.848800 24695 raft_consensus.cc:504] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.849565 24695 leader_election.cc:288] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [CANDIDATE]: Term 1 election: Requested vote from peers ef241f2a813846d9820141be016f56fc (127.16.48.2:41957), 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121)
I20260619 16:56:16.850117 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c0968817954965b905696546f111f0" candidate_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83781e3e6e384aba96a9209cf1d408ce"
I20260619 16:56:16.850184 24476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c0968817954965b905696546f111f0" candidate_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef241f2a813846d9820141be016f56fc"
I20260619 16:56:16.850358 24614 raft_consensus.cc:3037] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:16.850414 24476 raft_consensus.cc:3037] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:16.853001 24614 raft_consensus.cc:2454] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9f9c54a5f34434969092bbd5ba6b85 in term 1.
I20260619 16:56:16.853615 24265 leader_election.cc:302] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9f9c54a5f34434969092bbd5ba6b85, 83781e3e6e384aba96a9209cf1d408ce; no voters:
I20260619 16:56:16.853724 24476 raft_consensus.cc:2454] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9f9c54a5f34434969092bbd5ba6b85 in term 1.
I20260619 16:56:16.853972 24695 raft_consensus.cc:2781] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:16.854497 24695 raft_consensus.cc:686] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 LEADER]: Becoming Leader. State: Replica: 0d9f9c54a5f34434969092bbd5ba6b85, State: Running, Role: LEADER
I20260619 16:56:16.855099 24695 consensus_queue.cc:237] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } }
I20260619 16:56:16.858047 24175 catalog_manager.cc:5495] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1). New cstate: current_term: 1 leader_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9f9c54a5f34434969092bbd5ba6b85" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 46111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 } health_report { overall_health: UNKNOWN } } }
I20260619 16:56:16.904340 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:16.898121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.16.48.1:46111)
0619 16:56:16.898664 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:16.898668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:16.898684 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:56:16.898748 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:16.898752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:16.898811 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:16.898891 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:16.899422 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.900210 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.902197 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.902635 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.902737 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.903362 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:16.903473 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:16.903540 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:16.903800 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 16:56:16.903875 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":290,"threads_started":1}
I20260619 16:56:16.923616 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:16.915580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:47355 (local address 127.16.48.2:41957)
0619 16:56:16.915821 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:16.915828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:16.916573 (+ 745us) server_negotiation.cc:407] Connection header received
0619 16:56:16.916756 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:16.916763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:16.916874 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:16.917021 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:16.917583 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.918754 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.921110 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.921643 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.921781 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.922479 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:16.922610 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:16.922709 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:16.923091 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 16:56:16.923190 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 16:56:16.923885 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:16.915457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41957 (local address 127.16.48.1:47355)
0619 16:56:16.916422 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:16.916458 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:16.916608 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:16.917098 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:16.917102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:16.917113 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:56:16.917405 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:16.917413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.918993 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.919001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:16.920867 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:16.920890 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.921867 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.921876 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:16.922102 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.922204 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:56:16.922914 (+ 710us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:56:16.922928 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 16:56:16.923055 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 16:56:16.923258 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":565,"threads_started":1}
I20260619 16:56:16.935266 24614 raft_consensus.cc:1261] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Refusing update from remote peer 0d9f9c54a5f34434969092bbd5ba6b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:16.935352 24476 raft_consensus.cc:1261] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Refusing update from remote peer 0d9f9c54a5f34434969092bbd5ba6b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:16.935923 24783 consensus_queue.cc:1035] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83781e3e6e384aba96a9209cf1d408ce" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:16.936682 24695 consensus_queue.cc:1035] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef241f2a813846d9820141be016f56fc" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:16.991178 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:16.976413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:37033 (local address 127.16.48.1:46111)
0619 16:56:16.976589 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:16.976596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:16.985587 (+ 8991us) server_negotiation.cc:407] Connection header received
0619 16:56:16.985861 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:16.985867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:16.985951 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:16.986067 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:16.986648 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.987389 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.989055 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.989633 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.989791 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.990299 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:16.990485 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:16.990571 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:16.990781 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 16:56:16.990868 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:56:16.992017 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:16.976284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:46111 (local address 127.16.48.2:37033)
0619 16:56:16.985457 (+ 9173us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:16.985484 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:16.985689 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:16.986092 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:16.986096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:16.986109 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:56:16.986466 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:16.986477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.987561 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.987567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:16.988879 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:16.988893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:16.989841 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:16.989849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:16.990036 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:16.990086 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:56:16.990619 (+ 533us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:56:16.990647 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:56:16.991416 (+ 769us) client_negotiation.cc:240] Negotiation successful
0619 16:56:16.991587 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8979,"thread_start_us":8785,"threads_started":1}
I20260619 16:56:17.032724 24175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:17.033052 24175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:17.036675 24175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de1357d639d14c5e8904649573399a9c: Sending DeleteTablet for 3 replicas of tablet 45c0968817954965b905696546f111f0
I20260619 16:56:17.037621 24314 tablet_service.cc:1514] Processing DeleteTablet for tablet 45c0968817954965b905696546f111f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:17 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:34190
I20260619 16:56:17.037763 24454 tablet_service.cc:1514] Processing DeleteTablet for tablet 45c0968817954965b905696546f111f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:17 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:44142
I20260619 16:56:17.037959 24755 tablet_replica.cc:330] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: stopping tablet replica
I20260619 16:56:17.037949 24594 tablet_service.cc:1514] Processing DeleteTablet for tablet 45c0968817954965b905696546f111f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:17 UTC) from {username='oryx', principal='oryx/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:36012
I20260619 16:56:17.038129 24757 tablet_replica.cc:330] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: stopping tablet replica
I20260619 16:56:17.038133 24755 raft_consensus.cc:2229] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:56:17.038232 24756 tablet_replica.cc:330] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: stopping tablet replica
I20260619 16:56:17.038311 24757 raft_consensus.cc:2229] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:17.038401 24756 raft_consensus.cc:2229] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:17.038478 24755 raft_consensus.cc:2258] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:17.038513 24757 raft_consensus.cc:2258] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:17.038616 24756 raft_consensus.cc:2258] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:17.039481 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 24248
I20260619 16:56:17.040030 24755 ts_tablet_manager.cc:1905] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:17.040036 24757 ts_tablet_manager.cc:1905] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:17.047758 24757 ts_tablet_manager.cc:1918] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:56:17.047935 24757 log.cc:1196] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/45c0968817954965b905696546f111f0
I20260619 16:56:17.049204 24756 ts_tablet_manager.cc:1905] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:17.051479 24755 ts_tablet_manager.cc:1918] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:56:17.051769 24755 log.cc:1196] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/45c0968817954965b905696546f111f0
I20260619 16:56:17.052429 24755 ts_tablet_manager.cc:1939] T 45c0968817954965b905696546f111f0 P 0d9f9c54a5f34434969092bbd5ba6b85: Deleting consensus metadata
I20260619 16:56:17.054123 24757 ts_tablet_manager.cc:1939] T 45c0968817954965b905696546f111f0 P ef241f2a813846d9820141be016f56fc: Deleting consensus metadata
I20260619 16:56:17.054848 24163 catalog_manager.cc:4841] TS 0d9f9c54a5f34434969092bbd5ba6b85 (127.16.48.1:46111): tablet 45c0968817954965b905696546f111f0 (table test-table [id=f4278dc69ef34581bad21d3329710d9b]) successfully deleted
I20260619 16:56:17.055478 24164 catalog_manager.cc:4841] TS ef241f2a813846d9820141be016f56fc (127.16.48.2:41957): tablet 45c0968817954965b905696546f111f0 (table test-table [id=f4278dc69ef34581bad21d3329710d9b]) successfully deleted
I20260619 16:56:17.056965 24756 ts_tablet_manager.cc:1918] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:56:17.057132 24756 log.cc:1196] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/45c0968817954965b905696546f111f0
I20260619 16:56:17.057588 24756 ts_tablet_manager.cc:1939] T 45c0968817954965b905696546f111f0 P 83781e3e6e384aba96a9209cf1d408ce: Deleting consensus metadata
I20260619 16:56:17.058815 24162 catalog_manager.cc:4841] TS 83781e3e6e384aba96a9209cf1d408ce (127.16.48.3:35121): tablet 45c0968817954965b905696546f111f0 (table test-table [id=f4278dc69ef34581bad21d3329710d9b]) successfully deleted
Jun 19 16:56:17 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.1@KRBTEST.COM
I20260619 16:56:17.071316 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:17.041352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.16.48.1:46111)
0619 16:56:17.041650 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:17.041656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:17.053858 (+ 12202us) server_negotiation.cc:407] Connection header received
0619 16:56:17.055236 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:17.055242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:17.055348 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:17.055437 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:17.057039 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:17.057936 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:17.059470 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:17.059922 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:17.060094 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:17.064994 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:17.065011 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:17.065028 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:17.065073 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:17.068204 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:17.069040 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:17.069048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:17.069051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:17.069205 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:17.069662 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:17.069675 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:17.069678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:17.070045 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:17.070376 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:17.070698 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 16:56:17.070938 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260619 16:56:17.072055 24379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:17.199574 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 24248
I20260619 16:56:17.223727 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 24388
Jun 19 16:56:17 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.2@KRBTEST.COM
I20260619 16:56:17.239843 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:17.224637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.16.48.2:41957)
0619 16:56:17.224804 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:17.224810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:17.224829 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:56:17.224921 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:17.224926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:17.225009 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:17.225135 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:17.226364 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:17.227114 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:17.228590 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:17.229007 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:17.229163 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:17.233741 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:17.233757 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:17.233776 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:17.233824 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:17.236761 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:17.237579 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:17.237589 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:17.237593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:17.237761 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:17.238195 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:17.238200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:17.238203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:17.238576 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:17.238899 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:17.239220 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:56:17.239460 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:56:17.240641 24519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:17.386222 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 24388
I20260619 16:56:17.413232 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 24528
Jun 19 16:56:17 dist-test-slave-3ffd krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.48.3@KRBTEST.COM
I20260619 16:56:17.429948 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:17.413987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.16.48.3:35121)
0619 16:56:17.414513 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:17.414517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:17.414533 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:56:17.414605 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:17.414608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:17.414673 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:17.414767 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:17.416086 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:17.416921 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:17.418257 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:17.418713 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:17.418828 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:17.423718 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:17.423736 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:17.423753 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:17.423797 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:17.427032 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:17.427855 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:17.427865 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:17.427867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:17.428039 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:17.428541 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:17.428546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:17.428550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:17.428910 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:17.429141 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:17.429490 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 16:56:17.429676 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":293,"threads_started":1}
I20260619 16:56:17.430676 24659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:17.555271 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 24528
I20260619 16:56:17.578420 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 24145
I20260619 16:56:17.579767 24217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:17.700434 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 24145
2026-06-19T16:56:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6334 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:56:17 dist-test-slave-3ffd krb5kdc[24804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:56:17 dist-test-slave-3ffd krb5kdc[24804](info): set up 2 sockets
Jun 19 16:56:17 dist-test-slave-3ffd krb5kdc[24804](info): commencing operation
krb5kdc: starting...
W20260619 16:56:18.813513 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:56:18 dist-test-slave-3ffd krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:56:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:56:19.060264 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:45221
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:45221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:19.407891 24817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:19.408373 24817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:19.408547 24817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:19.416304 24817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:19.416426 24817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:19.416532 24817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:19.416642 24817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:19.416721 24817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:19.427577 24817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:45221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:45221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:19.429029 24817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:19.430558 24817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:19.443647 24825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:19.443740 24823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:19.443701 24822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:19.444990 24817 server_base.cc:1034] running on GCE node
I20260619 16:56:19.445956 24817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:19.447670 24817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:19.448947 24817 hybrid_clock.cc:648] HybridClock initialized: now 1781888179448864 us; error 75 us; skew 500 ppm
I20260619 16:56:19.449350 24817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T16:56:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1818 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:56:19 dist-test-slave-3ffd krb5kdc[24831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:56:19 dist-test-slave-3ffd krb5kdc[24831](info): set up 2 sockets
Jun 19 16:56:19 dist-test-slave-3ffd krb5kdc[24831](info): commencing operation
krb5kdc: starting...
W20260619 16:56:20.629630 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:56:20 dist-test-slave-3ffd krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:56:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:56:20.882513 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46429
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:46429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:21.232889 24844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:21.233227 24844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:21.233395 24844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:21.241156 24844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:21.241287 24844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:21.241405 24844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:21.241520 24844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:21.241607 24844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:21.251961 24844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:46429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:21.253463 24844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:21.254945 24844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:21.268725 24849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:21.268697 24850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:21.268697 24852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:21.269582 24844 server_base.cc:1034] running on GCE node
I20260619 16:56:21.270555 24844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:21.272207 24844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:21.273500 24844 hybrid_clock.cc:648] HybridClock initialized: now 1781888181273445 us; error 46 us; skew 500 ppm
I20260619 16:56:21.273905 24844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:21 dist-test-slave-3ffd krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888181, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:21.283967 24844 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:56:21.285734 24844 webserver.cc:460] Webserver started at http://127.16.48.62:44151/ using document root <none> and password file <none>
I20260619 16:56:21.286520 24844 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:21.286671 24844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:21.287021 24844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:21.290256 24844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "738f86773eb64a9196523c82b392bd93"
format_stamp: "Formatted at 2026-06-19 16:56:21 on dist-test-slave-3ffd"
server_key: "8305ba0ce3a3a5e8cb11815edb927ece"
server_key_iv: "afd4d7fd4ec1a396f3ff27dcd33700c8"
server_key_version: "encryptionkey@0"
I20260619 16:56:21.291107 24844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "738f86773eb64a9196523c82b392bd93"
format_stamp: "Formatted at 2026-06-19 16:56:21 on dist-test-slave-3ffd"
server_key: "8305ba0ce3a3a5e8cb11815edb927ece"
server_key_iv: "afd4d7fd4ec1a396f3ff27dcd33700c8"
server_key_version: "encryptionkey@0"
I20260619 16:56:21.297168 24844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 16:56:21.301206 24859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:21.302826 24844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:56:21.303047 24844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "738f86773eb64a9196523c82b392bd93"
format_stamp: "Formatted at 2026-06-19 16:56:21 on dist-test-slave-3ffd"
server_key: "8305ba0ce3a3a5e8cb11815edb927ece"
server_key_iv: "afd4d7fd4ec1a396f3ff27dcd33700c8"
server_key_version: "encryptionkey@0"
I20260619 16:56:21.303262 24844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:21.328035 24844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:21.332856 24844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:21.333179 24844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:21.353092 24844 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:46429
I20260619 16:56:21.353116 24910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:46429 every 8 connection(s)
I20260619 16:56:21.354734 24844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:56:21.357190 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 24844
I20260619 16:56:21.357419 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:56:21.358246 16576 external_mini_cluster.cc:1428] Setting key a92f9026c9898fc2e13bab74f1b854e4
I20260619 16:56:21.360297 24911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:56:21 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:56:21.375414 24911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: Bootstrap starting.
I20260619 16:56:21.379496 24911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:21.380949 24911 log.cc:824] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:21.384800 24911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: No bootstrap required, opened a new log
I20260619 16:56:21.385270 24914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:21.360640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.16.48.62:46429)
0619 16:56:21.361518 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:21.361534 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:21.361587 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:56:21.363799 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:21.363834 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:21.364511 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:21.365034 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:21.366874 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:21.369078 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:21.370526 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:21.371671 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:21.371891 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:21.376456 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:21.376466 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:21.376485 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:21.376557 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:21.380043 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:21.381179 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:21.381187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:21.381194 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:21.381374 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:21.381899 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:21.381904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:21.381907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:21.382768 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:21.382988 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:21.383543 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 16:56:21.384024 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":362,"threads_started":1}
I20260619 16:56:21.393594 24911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } }
I20260619 16:56:21.394068 24911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:21.394202 24911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738f86773eb64a9196523c82b392bd93, State: Initialized, Role: FOLLOWER
I20260619 16:56:21.394935 24911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } }
I20260619 16:56:21.395205 24911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:56:21.395344 24911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:56:21.395543 24911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:21.397934 24911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } }
I20260619 16:56:21.398552 24911 leader_election.cc:302] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 738f86773eb64a9196523c82b392bd93; no voters:
I20260619 16:56:21.399283 24911 leader_election.cc:288] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:56:21.399451 24916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:21.400353 24916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 1 LEADER]: Becoming Leader. State: Replica: 738f86773eb64a9196523c82b392bd93, State: Running, Role: LEADER
I20260619 16:56:21.401005 24916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } }
I20260619 16:56:21.401736 24911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:56:21.404394 24918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 738f86773eb64a9196523c82b392bd93. Latest consensus state: current_term: 1 leader_uuid: "738f86773eb64a9196523c82b392bd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } } }
I20260619 16:56:21.404901 24918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:21.405884 24917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "738f86773eb64a9196523c82b392bd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } } }
I20260619 16:56:21.406275 24917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:21.407894 24923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:56:21.413589 24923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:56:21.424768 24923 catalog_manager.cc:1347] Generated new cluster ID: c2eb84e4139748a8aa492c94739d5814
I20260619 16:56:21.424907 24923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:56:21.441191 24923 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:56:21.442615 24923 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:56:21.454895 24923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: Generated new TSK 0
I20260619 16:56:21.455915 24923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:56:21.512878 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46429
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:21.882696 24937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:21.883046 24937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:21.883158 24937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:21.892377 24937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:21.892515 24937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:21.892652 24937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:56:21.904661 24937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:21.906065 24937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:21.907544 24937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:21.922302 24945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:21.922389 24942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:21.922317 24943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:21.923166 24937 server_base.cc:1034] running on GCE node
I20260619 16:56:21.923939 24937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:21.925132 24937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:21.926426 24937 hybrid_clock.cc:648] HybridClock initialized: now 1781888181926357 us; error 72 us; skew 500 ppm
I20260619 16:56:21.926834 24937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:21 dist-test-slave-3ffd krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888181, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:21.930434 24937 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:56:21.932341 24937 webserver.cc:460] Webserver started at http://127.16.48.1:39585/ using document root <none> and password file <none>
I20260619 16:56:21.933116 24937 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:21.933233 24937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:21.933511 24937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:21.936786 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "b52d3d739b834fe5bb7ffa1b833b05ab"
format_stamp: "Formatted at 2026-06-19 16:56:21 on dist-test-slave-3ffd"
server_key: "0e3227489fd34dee1874688fee9bc92b"
server_key_iv: "ac19e3355199770fc8b22f2a444f3ab8"
server_key_version: "encryptionkey@0"
I20260619 16:56:21.937721 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "b52d3d739b834fe5bb7ffa1b833b05ab"
format_stamp: "Formatted at 2026-06-19 16:56:21 on dist-test-slave-3ffd"
server_key: "0e3227489fd34dee1874688fee9bc92b"
server_key_iv: "ac19e3355199770fc8b22f2a444f3ab8"
server_key_version: "encryptionkey@0"
I20260619 16:56:21.943610 24937 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260619 16:56:21.947683 24952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:21.949566 24937 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:56:21.949740 24937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "b52d3d739b834fe5bb7ffa1b833b05ab"
format_stamp: "Formatted at 2026-06-19 16:56:21 on dist-test-slave-3ffd"
server_key: "0e3227489fd34dee1874688fee9bc92b"
server_key_iv: "ac19e3355199770fc8b22f2a444f3ab8"
server_key_version: "encryptionkey@0"
I20260619 16:56:21.949931 24937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:21.964684 24937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:21.969777 24937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:21.970091 24937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:21.971297 24937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:21.973053 24937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:21.973138 24937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:21.973232 24937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:21.973277 24937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:22.010211 24937 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:39349
I20260619 16:56:22.010258 25064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:39349 every 8 connection(s)
I20260619 16:56:22.011653 24937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:56:22.021127 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 24937
I20260619 16:56:22.021341 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:56:22.021976 16576 external_mini_cluster.cc:1428] Setting key 24180d62b5f967c4325e42a5c4b1e301
Jun 19 16:56:22 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888181, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:56:22.035231 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:22.014660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:34771 (local address 127.16.48.62:46429)
0619 16:56:22.015125 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:22.015131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:22.016262 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 16:56:22.018242 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:22.018248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:22.018349 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:22.018488 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:22.020242 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:22.020970 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:22.023102 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:22.023536 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:22.023712 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:22.027575 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:22.027581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:22.027584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:22.027626 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:22.031098 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:22.032531 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:22.032610 (+ 79us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:22.032623 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:22.032798 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:22.033330 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:22.033335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:22.033338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:22.033630 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:22.033800 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:22.034594 (+ 794us) server_negotiation.cc:299] Negotiation successful
0619 16:56:22.034839 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":257,"threads_started":1}
I20260619 16:56:22.036615 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:22.015256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:34771)
0619 16:56:22.016014 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:22.016067 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:22.017899 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:22.018777 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:22.018787 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:22.019370 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:22.019995 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:22.020014 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:22.021179 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:22.021183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:22.022918 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:22.022938 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:22.023814 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:22.023825 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:22.024102 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:22.024949 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:22.024972 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:22.027279 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:22.031334 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:22.031348 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:22.031372 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:22.032314 (+ 942us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:22.032940 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:22.032946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:22.032949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:22.033185 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:22.033779 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:22.033787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:22.034194 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 16:56:22.034495 (+ 301us) client_negotiation.cc:240] Negotiation successful
0619 16:56:22.035099 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":308,"threads_started":1}
I20260619 16:56:22.038542 25065 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46429
I20260619 16:56:22.038916 25065 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:22.039912 25065 heartbeater.cc:511] Master 127.16.48.62:46429 requested a full tablet report, sending...
I20260619 16:56:22.042829 24876 ts_manager.cc:194] Registered new tserver with Master: b52d3d739b834fe5bb7ffa1b833b05ab (127.16.48.1:39349)
I20260619 16:56:22.045288 24876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:34771
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:56:22.080932 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46429
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:22.439673 25071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:22.439991 25071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:22.440196 25071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:22.448298 25071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:22.448433 25071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:22.448666 25071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:56:22.459697 25071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:22.461131 25071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:22.462522 25071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:22.477676 25076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:22.477686 25079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:22.477728 25077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:22.478865 25071 server_base.cc:1034] running on GCE node
I20260619 16:56:22.479591 25071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:22.480674 25071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:22.481891 25071 hybrid_clock.cc:648] HybridClock initialized: now 1781888182481821 us; error 65 us; skew 500 ppm
I20260619 16:56:22.482249 25071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:22 dist-test-slave-3ffd krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:22.485910 25071 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:56:22.487567 25071 webserver.cc:460] Webserver started at http://127.16.48.2:46515/ using document root <none> and password file <none>
I20260619 16:56:22.488430 25071 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:22.488588 25071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:22.488940 25071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:22.492194 25071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "dee483bf06ee49f599d2f3080fb2a9ef"
format_stamp: "Formatted at 2026-06-19 16:56:22 on dist-test-slave-3ffd"
server_key: "ce384c5d0ee8a3e0241c770cd001f25c"
server_key_iv: "893f77cbc55ca74cf3b4ab730c746ef5"
server_key_version: "encryptionkey@0"
I20260619 16:56:22.493099 25071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "dee483bf06ee49f599d2f3080fb2a9ef"
format_stamp: "Formatted at 2026-06-19 16:56:22 on dist-test-slave-3ffd"
server_key: "ce384c5d0ee8a3e0241c770cd001f25c"
server_key_iv: "893f77cbc55ca74cf3b4ab730c746ef5"
server_key_version: "encryptionkey@0"
I20260619 16:56:22.499135 25071 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 16:56:22.503438 25086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:22.505143 25071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:56:22.505378 25071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "dee483bf06ee49f599d2f3080fb2a9ef"
format_stamp: "Formatted at 2026-06-19 16:56:22 on dist-test-slave-3ffd"
server_key: "ce384c5d0ee8a3e0241c770cd001f25c"
server_key_iv: "893f77cbc55ca74cf3b4ab730c746ef5"
server_key_version: "encryptionkey@0"
I20260619 16:56:22.505602 25071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:22.520450 25071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:22.525187 25071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:22.525529 25071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:22.526580 25071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:22.528072 25071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:22.528231 25071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 16:56:22.528391 25071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:22.528472 25071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:22.564085 25071 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:37401
I20260619 16:56:22.564186 25198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:37401 every 8 connection(s)
I20260619 16:56:22.565703 25071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:56:22.575939 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 25071
I20260619 16:56:22.576084 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:56:22.576681 16576 external_mini_cluster.cc:1428] Setting key e412667724c289ca0e365d26fa2bd876
Jun 19 16:56:22 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:56:22.588303 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:22.568547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:54173 (local address 127.16.48.62:46429)
0619 16:56:22.568971 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:22.568976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:22.570094 (+ 1118us) server_negotiation.cc:407] Connection header received
0619 16:56:22.571990 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:22.571996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:22.572095 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:22.572289 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:22.573908 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:22.574632 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:22.576507 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:22.577022 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:22.577182 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:22.581119 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:22.581124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:22.581127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:22.581171 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:22.584063 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:22.585188 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:22.585203 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:22.585206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:22.585380 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:22.585979 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:22.585987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:22.585992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:22.586426 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:22.586615 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:22.587465 (+ 850us) server_negotiation.cc:299] Negotiation successful
0619 16:56:22.587766 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":217,"threads_started":1}
I20260619 16:56:22.589311 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:22.569172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:54173)
0619 16:56:22.569912 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:22.569952 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:22.571679 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:22.572547 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:22.572556 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:22.573105 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:22.573711 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:22.573729 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:22.574823 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:22.574826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:22.576343 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:22.576357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:22.577281 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:22.577289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:22.577574 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:22.578389 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:22.578421 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:22.580833 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:22.584364 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:22.584376 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:22.584401 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:22.584941 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:22.585585 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:22.585588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:22.585591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:22.585840 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:22.586644 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:22.586651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:22.587047 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 16:56:22.587319 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 16:56:22.587909 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
I20260619 16:56:22.591118 25199 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46429
I20260619 16:56:22.591495 25199 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:22.592415 25199 heartbeater.cc:511] Master 127.16.48.62:46429 requested a full tablet report, sending...
I20260619 16:56:22.594080 24876 ts_manager.cc:194] Registered new tserver with Master: dee483bf06ee49f599d2f3080fb2a9ef (127.16.48.2:37401)
I20260619 16:56:22.594951 24876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:54173
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:56:22.633293 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46429
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:22.992949 25205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:22.993305 25205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:22.993491 25205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:23.001052 25205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:23.001194 25205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:23.001462 25205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:56:23.012203 25205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:23.013708 25205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:23.015251 25205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:23.030048 25211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:23.029965 25213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:23.030092 25210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:23.030887 25205 server_base.cc:1034] running on GCE node
I20260619 16:56:23.031610 25205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:23.032871 25205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:23.034101 25205 hybrid_clock.cc:648] HybridClock initialized: now 1781888183034015 us; error 75 us; skew 500 ppm
I20260619 16:56:23.034519 25205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:23 dist-test-slave-3ffd krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888183, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:23.038538 25205 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:56:23.040341 25205 webserver.cc:460] Webserver started at http://127.16.48.3:44873/ using document root <none> and password file <none>
I20260619 16:56:23.041157 25205 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:23.041303 25205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:23.041661 25205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:23.044914 25205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "7ad77dab584d462f90dbdbcb8c493da3"
format_stamp: "Formatted at 2026-06-19 16:56:23 on dist-test-slave-3ffd"
server_key: "59dd51c04f0ebc9ccfd0bdf577ca983e"
server_key_iv: "0bb09e48cb37b4c235564c44af46554a"
server_key_version: "encryptionkey@0"
I20260619 16:56:23.045825 25205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "7ad77dab584d462f90dbdbcb8c493da3"
format_stamp: "Formatted at 2026-06-19 16:56:23 on dist-test-slave-3ffd"
server_key: "59dd51c04f0ebc9ccfd0bdf577ca983e"
server_key_iv: "0bb09e48cb37b4c235564c44af46554a"
server_key_version: "encryptionkey@0"
I20260619 16:56:23.049345 25065 heartbeater.cc:503] Master 127.16.48.62:46429 was elected leader, sending a full tablet report...
I20260619 16:56:23.051975 25205 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 16:56:23.056245 25220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:23.057847 25205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:23.058064 25205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "7ad77dab584d462f90dbdbcb8c493da3"
format_stamp: "Formatted at 2026-06-19 16:56:23 on dist-test-slave-3ffd"
server_key: "59dd51c04f0ebc9ccfd0bdf577ca983e"
server_key_iv: "0bb09e48cb37b4c235564c44af46554a"
server_key_version: "encryptionkey@0"
I20260619 16:56:23.058308 25205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:23.082487 25205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:23.087343 25205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:23.087692 25205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:23.089056 25205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:23.090586 25205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:23.090718 25205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:23.090874 25205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:23.090948 25205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:23.126406 25205 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:33851
I20260619 16:56:23.126466 25332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:33851 every 8 connection(s)
I20260619 16:56:23.127817 25205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:56:23.138151 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 25205
I20260619 16:56:23.138337 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:56:23.138898 16576 external_mini_cluster.cc:1428] Setting key 73f77bea652496b6e5fa97df5de0b214
Jun 19 16:56:23 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888183, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:56:23.151885 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:23.130704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:56267 (local address 127.16.48.62:46429)
0619 16:56:23.131193 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:23.131199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:23.132305 (+ 1106us) server_negotiation.cc:407] Connection header received
0619 16:56:23.134240 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:23.134245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:23.134320 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:23.134425 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:23.136358 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.137159 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.139637 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.140299 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.140477 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:23.145616 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:23.145622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:23.145625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:23.145690 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:23.148733 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:23.149602 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:23.149606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:23.149610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:23.149772 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:23.150199 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:23.150203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:23.150205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:23.150477 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:23.150591 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:23.151380 (+ 789us) server_negotiation.cc:299] Negotiation successful
0619 16:56:23.151580 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":227,"threads_started":1}
I20260619 16:56:23.153080 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:23.131235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:56267)
0619 16:56:23.132039 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:23.132082 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:23.133916 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:23.134809 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:23.134820 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:23.135388 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:23.136040 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:23.136059 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.137322 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.137326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:23.139410 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:23.139430 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.140751 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.140760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:23.141129 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:23.142060 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:23.142087 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:23.145338 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:23.148910 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:23.148919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:23.148934 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:23.149461 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:23.149882 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:23.149885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:23.149888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:23.150084 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:23.150593 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:23.150599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:23.150985 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 16:56:23.151259 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 16:56:23.151716 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":320,"threads_started":1}
I20260619 16:56:23.154939 25333 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46429
I20260619 16:56:23.155331 25333 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:23.156313 25333 heartbeater.cc:511] Master 127.16.48.62:46429 requested a full tablet report, sending...
I20260619 16:56:23.157923 24876 ts_manager.cc:194] Registered new tserver with Master: 7ad77dab584d462f90dbdbcb8c493da3 (127.16.48.3:33851)
I20260619 16:56:23.158812 24876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:56267
I20260619 16:56:23.168774 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab.
Entry for principal oryx/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab.
WARNING: no policy specified for oryx/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab.
Entry for principal oryx/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab.
WARNING: no policy specified for oryx/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab.
Entry for principal oryx/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab.
I20260619 16:56:23.323920 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 24937
Jun 19 16:56:23 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:56:23.342629 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:23.325362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.16.48.1:39349)
0619 16:56:23.326098 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:23.326111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:23.326139 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:56:23.326204 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:23.326211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:23.326444 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:23.326591 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:23.327957 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.329153 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.330432 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.331057 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.331178 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:23.335660 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:23.335676 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:23.335694 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:23.335741 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:23.338763 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:23.339878 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:23.339887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:23.339890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:23.340058 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:23.340667 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:23.340672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:23.340674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:23.341251 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:23.341527 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:23.341987 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 16:56:23.342257 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":407,"threads_started":1}
I20260619 16:56:23.344058 25060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:23.429589 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 24937
I20260619 16:56:23.451367 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25071
Jun 19 16:56:23 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:56:23.476397 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:23.452559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.16.48.2:37401)
0619 16:56:23.453612 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:23.453625 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:23.453660 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:56:23.453762 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:23.453771 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:23.454171 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:23.454415 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:23.456645 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.458318 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.460528 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.461803 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.461969 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:23.467685 (+ 5716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:23.467707 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:23.467730 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:23.467786 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:23.472258 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:23.473257 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:23.473270 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:23.473277 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:23.473491 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:23.474046 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:23.474055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:23.474061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:23.474639 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:23.475013 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:23.475439 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 16:56:23.475801 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":498,"threads_started":1}
I20260619 16:56:23.477892 25194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:23.561690 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 25071
I20260619 16:56:23.578650 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25205
Jun 19 16:56:23 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:56:23.596777 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:23.579792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.16.48.3:33851)
0619 16:56:23.580384 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:23.580391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:23.580424 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:56:23.580484 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:23.580491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:23.580720 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:23.580914 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:23.582320 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.583350 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.584864 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:23.585510 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:23.585691 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:23.590484 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:23.590501 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:23.590517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:23.590566 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:23.593585 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:23.594435 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:23.594443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:23.594446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:23.594584 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:23.595029 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:23.595035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:23.595038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:23.595426 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:23.595754 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:23.596077 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 16:56:23.596392 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":257,"threads_started":1}
I20260619 16:56:23.598153 25328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:23.683104 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 25205
I20260619 16:56:23.699417 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 24844
I20260619 16:56:23.700624 24906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:23.802125 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 24844
I20260619 16:56:23.823272 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46429
--webserver_interface=127.16.48.62
--webserver_port=44151
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:46429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:24.172510 25351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:24.172856 25351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:24.172945 25351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:24.180435 25351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:24.180567 25351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:24.180619 25351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:24.180678 25351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:24.180722 25351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:24.191329 25351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:46429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=44151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:24.192798 25351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:24.194203 25351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:24.207468 25356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:24.207434 25357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:24.207367 25359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:24.208721 25351 server_base.cc:1034] running on GCE node
I20260619 16:56:24.209599 25351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:24.211182 25351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:24.212417 25351 hybrid_clock.cc:648] HybridClock initialized: now 1781888184212361 us; error 45 us; skew 500 ppm
I20260619 16:56:24.212803 25351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:24 dist-test-slave-3ffd krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888184, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:24.216380 25351 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:56:24.218102 25351 webserver.cc:460] Webserver started at http://127.16.48.62:44151/ using document root <none> and password file <none>
I20260619 16:56:24.218988 25351 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:24.219161 25351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:24.225390 25351 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:56:24.228826 25366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:24.230499 25351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:24.230726 25351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "738f86773eb64a9196523c82b392bd93"
format_stamp: "Formatted at 2026-06-19 16:56:21 on dist-test-slave-3ffd"
server_key: "8305ba0ce3a3a5e8cb11815edb927ece"
server_key_iv: "afd4d7fd4ec1a396f3ff27dcd33700c8"
server_key_version: "encryptionkey@0"
I20260619 16:56:24.231848 25351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:24.247953 25351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:24.252458 25351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:24.252755 25351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:24.272921 25351 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:46429
I20260619 16:56:24.272939 25417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:46429 every 8 connection(s)
I20260619 16:56:24.274593 25351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:56:24.277274 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 25351
I20260619 16:56:24.278277 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:39349
--local_ip_for_outbound_sockets=127.16.48.1
--tserver_master_addrs=127.16.48.62:46429
--webserver_port=39585
--webserver_interface=127.16.48.1
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.48.1
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:56:24.284713 25418 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:56:24.289168 25418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: Bootstrap starting.
I20260619 16:56:24.310556 25418 log.cc:824] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:24.321316 25418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:56:24.321943 25418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: Bootstrap complete.
I20260619 16:56:24.331568 25418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } }
I20260619 16:56:24.332628 25418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738f86773eb64a9196523c82b392bd93, State: Initialized, Role: FOLLOWER
I20260619 16:56:24.333312 25418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } }
I20260619 16:56:24.333583 25418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:56:24.333729 25418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:56:24.333925 25418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:56:24.338609 25418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } }
I20260619 16:56:24.339236 25418 leader_election.cc:302] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 738f86773eb64a9196523c82b392bd93; no voters:
I20260619 16:56:24.339927 25418 leader_election.cc:288] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:56:24.340085 25422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:56:24.342799 25418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:56:24.344445 25422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [term 2 LEADER]: Becoming Leader. State: Replica: 738f86773eb64a9196523c82b392bd93, State: Running, Role: LEADER
I20260619 16:56:24.345326 25422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } }
I20260619 16:56:24.352428 25423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "738f86773eb64a9196523c82b392bd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } } }
I20260619 16:56:24.352926 25423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:24.351599 25424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 738f86773eb64a9196523c82b392bd93. Latest consensus state: current_term: 2 leader_uuid: "738f86773eb64a9196523c82b392bd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f86773eb64a9196523c82b392bd93" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46429 } } }
I20260619 16:56:24.353566 25424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:24.359932 25433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:56:24.365535 25433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:56:24.367192 25433 catalog_manager.cc:1259] Loaded cluster ID: c2eb84e4139748a8aa492c94739d5814
I20260619 16:56:24.367312 25433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:56:24.371498 25433 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:56:24.373243 25433 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 738f86773eb64a9196523c82b392bd93: Loaded TSK: 0
I20260619 16:56:24.375079 25433 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:56:24.657256 25420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:24.657640 25420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:24.657816 25420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:24.665699 25420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:24.665875 25420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:24.666055 25420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:56:24.676437 25420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.1.keytab
--principal=oryx/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:39349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=39585
--tserver_master_addrs=127.16.48.62:46429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:24.677896 25420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:24.679373 25420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:24.696461 25446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:24.698746 25448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:24.698663 25445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:24.702940 25420 server_base.cc:1034] running on GCE node
I20260619 16:56:24.703835 25420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:24.705087 25420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:24.706390 25420 hybrid_clock.cc:648] HybridClock initialized: now 1781888184706318 us; error 75 us; skew 500 ppm
I20260619 16:56:24.706843 25420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:24 dist-test-slave-3ffd krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888184, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:24.710795 25420 init.cc:377] Logged in from keytab as oryx/127.16.48.1@KRBTEST.COM (short username oryx)
I20260619 16:56:24.712967 25420 webserver.cc:460] Webserver started at http://127.16.48.1:39585/ using document root <none> and password file <none>
I20260619 16:56:24.713843 25420 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:24.714036 25420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:24.720525 25420 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:56:24.723798 25455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:24.725682 25420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:56:24.725894 25420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "b52d3d739b834fe5bb7ffa1b833b05ab"
format_stamp: "Formatted at 2026-06-19 16:56:21 on dist-test-slave-3ffd"
server_key: "0e3227489fd34dee1874688fee9bc92b"
server_key_iv: "ac19e3355199770fc8b22f2a444f3ab8"
server_key_version: "encryptionkey@0"
I20260619 16:56:24.727015 25420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:24.748939 25420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:24.753841 25420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:24.754246 25420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:24.755359 25420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:24.757001 25420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:24.757107 25420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:24.757280 25420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:24.757386 25420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:24.793390 25420 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:39349
I20260619 16:56:24.793506 25567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:39349 every 8 connection(s)
I20260619 16:56:24.794999 25420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:56:24.797078 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 25420
I20260619 16:56:24.797884 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:37401
--local_ip_for_outbound_sockets=127.16.48.2
--tserver_master_addrs=127.16.48.62:46429
--webserver_port=46515
--webserver_interface=127.16.48.2
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.48.2
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:56:24 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:24 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:24.819746 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:24.798858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:56455)
0619 16:56:24.800699 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:24.800750 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:24.802789 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:24.804812 (+ 2023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:24.804828 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:24.805514 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:24.806152 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:24.806175 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.809111 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.809117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:24.810566 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:24.810585 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.812576 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.812590 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:24.812898 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:24.813862 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:24.813895 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:24.818475 (+ 4580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1560,"thread_start_us":1346,"threads_started":1}
W20260619 16:56:24.820633 25568 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:46429 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.48.62:46429: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
W20260619 16:56:24.820870 25570 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:24.798798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:56455 (local address 127.16.48.62:46429)
0619 16:56:24.799594 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:24.799630 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:24.800923 (+ 1293us) server_negotiation.cc:407] Connection header received
0619 16:56:24.803511 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:24.803527 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:24.804060 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:24.804538 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:24.806435 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.808771 (+ 2336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.810806 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.812212 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.812454 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:24.819167 (+ 6713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:56455: BlockingRecv error: recv got EOF from 127.16.48.1:56455 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":847,"threads_started":1}
Jun 19 16:56:24 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:24 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:24.830309 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:24.821349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:48945)
0619 16:56:24.821559 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:24.821581 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:24.821743 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:24.822349 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:24.822357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:24.822764 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:24.822914 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:24.822923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.824208 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.824218 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:24.825153 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:24.825189 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.825953 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.825961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:24.826159 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:24.826886 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:24.826905 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:24.829969 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":32}
W20260619 16:56:24.830693 25573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:24.821476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:48945 (local address 127.16.48.62:46429)
0619 16:56:24.822021 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:24.822025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:24.822046 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:56:24.822100 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:24.822103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:24.822174 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:24.822315 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:24.823135 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.823976 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.825352 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.825777 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.825921 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:24.830309 (+ 4388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:48945: BlockingRecv error: recv got EOF from 127.16.48.1:48945 (error 108)
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":295,"threads_started":1}
Jun 19 16:56:24 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:24 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:24.841140 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:24.831381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:33567)
0619 16:56:24.831542 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:24.831561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:24.831674 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:24.832052 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:24.832055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:24.832432 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:24.832600 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:24.832609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.834124 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.834129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:24.835086 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:24.835098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.835886 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.835893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:24.836107 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:24.836894 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:24.836916 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:24.840797 (+ 3881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:56:24.841302 25573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:24.831512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:33567 (local address 127.16.48.62:46429)
0619 16:56:24.831688 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:24.831694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:24.831716 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:56:24.831802 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:24.831808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:24.831908 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:24.832006 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:24.832766 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.833926 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.835310 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:24.835724 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:24.835826 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:24.841046 (+ 5220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:33567: BlockingRecv error: recv got EOF from 127.16.48.1:33567 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 16:56:24.841529 25568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:56:25.154487 25572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:25.154873 25572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:25.155056 25572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:25.162645 25572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:25.162864 25572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:25.163072 25572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:56:25.173267 25572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.2.keytab
--principal=oryx/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:37401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=46515
--tserver_master_addrs=127.16.48.62:46429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:25.174799 25572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:25.176350 25572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:25.190922 25581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:25.191167 25578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:25.191143 25579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:25.192550 25572 server_base.cc:1034] running on GCE node
I20260619 16:56:25.193295 25572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:25.194365 25572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:25.195626 25572 hybrid_clock.cc:648] HybridClock initialized: now 1781888185195581 us; error 52 us; skew 500 ppm
I20260619 16:56:25.195982 25572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888185, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:25.199383 25572 init.cc:377] Logged in from keytab as oryx/127.16.48.2@KRBTEST.COM (short username oryx)
I20260619 16:56:25.201025 25572 webserver.cc:460] Webserver started at http://127.16.48.2:46515/ using document root <none> and password file <none>
I20260619 16:56:25.201761 25572 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:25.201879 25572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:25.208096 25572 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:56:25.211503 25588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:25.213207 25572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:56:25.213416 25572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "dee483bf06ee49f599d2f3080fb2a9ef"
format_stamp: "Formatted at 2026-06-19 16:56:22 on dist-test-slave-3ffd"
server_key: "ce384c5d0ee8a3e0241c770cd001f25c"
server_key_iv: "893f77cbc55ca74cf3b4ab730c746ef5"
server_key_version: "encryptionkey@0"
I20260619 16:56:25.214435 25572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:25.231839 25572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:25.236752 25572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:25.237051 25572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:25.238104 25572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:25.239547 25572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:25.239655 25572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:25.239871 25572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:25.239946 25572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:25.275286 25572 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:37401
I20260619 16:56:25.275336 25700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:37401 every 8 connection(s)
I20260619 16:56:25.276981 25572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:56:25.286208 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 25572
I20260619 16:56:25.286943 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:33851
--local_ip_for_outbound_sockets=127.16.48.3
--tserver_master_addrs=127.16.48.62:46429
--webserver_port=44873
--webserver_interface=127.16.48.3
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.48.3
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:56:25.295809 25573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:25.279956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:57249 (local address 127.16.48.62:46429)
0619 16:56:25.280210 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:25.280216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:25.281614 (+ 1398us) server_negotiation.cc:407] Connection header received
0619 16:56:25.284280 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:25.284285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:25.284382 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:25.284511 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:25.286238 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.287359 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.289138 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.289876 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.290032 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.295527 (+ 5495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:57249: BlockingRecv error: recv got EOF from 127.16.48.2:57249 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:56:25.296651 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:25.280523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:57249)
0619 16:56:25.281391 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:25.281454 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:25.283888 (+ 2434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:25.284809 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:25.284818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:25.285384 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:25.286018 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.286037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.287538 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.287542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.288952 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.288966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.290124 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.290129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.290358 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.291135 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:25.291160 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:25.295325 (+ 4165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":324,"threads_started":1}
W20260619 16:56:25.297453 25701 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:46429 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.48.62:46429: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:25.306249 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:25.298098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:57125)
0619 16:56:25.298259 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:25.298286 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:25.298414 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:25.298801 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:25.298805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:25.299174 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:25.299310 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.299318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.300368 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.300374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.301296 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.301307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.302079 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.302085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.302278 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.302975 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:25.302993 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:25.305931 (+ 2938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 16:56:25.306694 25573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:25.298301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:57125 (local address 127.16.48.62:46429)
0619 16:56:25.298513 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:25.298519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:25.298535 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:56:25.298597 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:25.298601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:25.298664 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:25.298778 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:25.299460 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.300226 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.301451 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.301915 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.302062 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.306174 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:57125: BlockingRecv error: recv got EOF from 127.16.48.2:57125 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:25.316285 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:25.306947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:38693)
0619 16:56:25.307129 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:25.307178 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:25.307287 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:25.307869 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:25.307872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:25.308226 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:25.308380 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.308389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.309983 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.309986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.310893 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.310903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.311877 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.311880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.312042 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.312762 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:25.312781 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:25.315922 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 16:56:25.316701 25573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:25.307281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:38693 (local address 127.16.48.62:46429)
0619 16:56:25.307494 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:25.307501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:25.307523 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:56:25.307594 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:25.307603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:25.307710 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:25.307842 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:25.308537 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.309827 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.311040 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.311728 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.311860 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.316247 (+ 4387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:38693: BlockingRecv error: recv got EOF from 127.16.48.2:38693 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 16:56:25.316709 25701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:56:25.644150 25704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:25.644459 25704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:25.644657 25704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:25.652570 25704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:25.652693 25704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:25.652884 25704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:56:25.662979 25704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/krb5kdc/oryx_127.16.48.3.keytab
--principal=oryx/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:33851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=44873
--tserver_master_addrs=127.16.48.62:46429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:25.664431 25704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:25.665906 25704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:25.681891 25709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:25.681835 25710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:25.681835 25712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:25.682479 25704 server_base.cc:1034] running on GCE node
I20260619 16:56:25.683130 25704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:25.684237 25704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:25.685478 25704 hybrid_clock.cc:648] HybridClock initialized: now 1781888185685431 us; error 53 us; skew 500 ppm
I20260619 16:56:25.685822 25704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888185, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:25.689327 25704 init.cc:377] Logged in from keytab as oryx/127.16.48.3@KRBTEST.COM (short username oryx)
I20260619 16:56:25.691080 25704 webserver.cc:460] Webserver started at http://127.16.48.3:44873/ using document root <none> and password file <none>
I20260619 16:56:25.691969 25704 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:25.692133 25704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:25.698539 25704 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:56:25.701756 25719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:25.703368 25704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:25.703572 25704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "7ad77dab584d462f90dbdbcb8c493da3"
format_stamp: "Formatted at 2026-06-19 16:56:23 on dist-test-slave-3ffd"
server_key: "59dd51c04f0ebc9ccfd0bdf577ca983e"
server_key_iv: "0bb09e48cb37b4c235564c44af46554a"
server_key_version: "encryptionkey@0"
I20260619 16:56:25.704754 25704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:25.726203 25704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:25.730875 25704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:25.731230 25704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:25.732575 25704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:25.734091 25704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:25.734184 25704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:25.734356 25704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:25.734467 25704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:25.771186 25704 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:33851
I20260619 16:56:25.771248 25831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:33851 every 8 connection(s)
I20260619 16:56:25.772821 25704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:56:25.773447 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 25704
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:25.792263 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:25.774992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.16.48.62:46429)
0619 16:56:25.775170 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:25.775175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:25.775322 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:56:25.775533 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:25.775538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:25.775649 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:25.775773 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:25.777600 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.778320 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.779827 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.780580 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.780724 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.783864 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:25.783874 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:25.783897 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:25.783950 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:25.788324 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:25.789285 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:25.789294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:25.789302 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:25.789546 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:25.790003 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:25.790010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:25.790013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:25.790649 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:25.790992 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:25.791457 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 16:56:25.791905 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 16:56:25.792536 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:25.776170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:58969 (local address 127.16.48.62:46429)
0619 16:56:25.776665 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:25.776672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:25.778506 (+ 1834us) server_negotiation.cc:407] Connection header received
0619 16:56:25.779678 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:25.779683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:25.779771 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:25.779880 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:25.781806 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.783003 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.785614 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.786196 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.786304 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.792222 (+ 5918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:58969: BlockingRecv error: recv got EOF from 127.16.48.3:58969 (error 108)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":259,"threads_started":1}
I20260619 16:56:25.793443 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:25.776784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:58969)
0619 16:56:25.777643 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:25.777691 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:25.779388 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:25.780235 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:25.780249 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:25.781019 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:25.781624 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.781643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.783196 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.783201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.785242 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.785261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.786476 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.786486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.786854 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.787565 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:25.787614 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:25.791845 (+ 4231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":357,"threads_started":1}
W20260619 16:56:25.794287 25832 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:46429 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.48.62:46429: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:25.805953 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:25.795536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:60747)
0619 16:56:25.795970 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:25.795987 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:25.796168 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:25.796682 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:25.796690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:25.797179 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:25.797394 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.797407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.798471 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.798476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.799968 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.799984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.801044 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.801050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.801204 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.802002 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:25.802025 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:25.805590 (+ 3565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 16:56:25.806334 25573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:25.795084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:60747 (local address 127.16.48.62:46429)
0619 16:56:25.795273 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:25.795278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:25.796070 (+ 792us) server_negotiation.cc:407] Connection header received
0619 16:56:25.796305 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:25.796310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:25.796385 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:25.796512 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:25.797594 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.798293 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.800380 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.800847 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.800966 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.805885 (+ 4919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:60747: BlockingRecv error: recv got EOF from 127.16.48.3:60747 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:25.816020 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:25.806851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:47203)
0619 16:56:25.807084 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:25.807108 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:25.807267 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:25.807725 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:25.807730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:25.808291 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:25.808534 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.808572 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.810042 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.810047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.811031 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.811044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.811777 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.811783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.811989 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.812731 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:25.812752 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:25.815779 (+ 3027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 16:56:25.816267 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:25.806979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:47203 (local address 127.16.48.62:46429)
0619 16:56:25.807239 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:25.807244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:25.807263 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:56:25.807424 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:25.807431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:25.807523 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:25.807639 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:25.808990 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.809902 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.811187 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.811590 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.811691 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.816013 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:47203: BlockingRecv error: recv got EOF from 127.16.48.3:47203 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
W20260619 16:56:25.816447 25832 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:25 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:25.851501 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:25.842327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:38099)
0619 16:56:25.842890 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:25.842905 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:25.843074 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:25.843493 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:25.843498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:25.843946 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:25.844227 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.844240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.845543 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.845549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.846650 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:25.846672 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.847422 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.847429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:25.847618 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.848274 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:25.848293 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:25.851260 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":324,"threads_started":1}
W20260619 16:56:25.851799 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:25.842437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:38099 (local address 127.16.48.62:46429)
0619 16:56:25.842733 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:25.842741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:25.843000 (+ 259us) server_negotiation.cc:407] Connection header received
0619 16:56:25.843271 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:25.843275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:25.843339 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:25.843451 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:25.844388 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.845394 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.846798 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:25.847222 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:25.847360 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:25.851464 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:38099: BlockingRecv error: recv got EOF from 127.16.48.1:38099 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 19 16:56:26 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:26 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:26.331990 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:26.317289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:55721)
0619 16:56:26.317871 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:26.317888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:26.318069 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:26.318436 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:26.318441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:26.319024 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:26.319234 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:26.319244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.320328 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.320334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:26.321334 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:26.321346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.322169 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.322177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:26.322352 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:26.328714 (+ 6362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:26.328737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:26.331757 (+ 3020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":340,"threads_started":1}
W20260619 16:56:26.332300 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:26.317417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:55721 (local address 127.16.48.62:46429)
0619 16:56:26.317579 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:26.317583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:26.318000 (+ 417us) server_negotiation.cc:407] Connection header received
0619 16:56:26.318205 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:26.318210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:26.318289 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:26.318404 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:26.319390 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.320106 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.321500 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.321943 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.322102 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:26.332028 (+ 9926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:55721: BlockingRecv error: recv got EOF from 127.16.48.2:55721 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 16:56:26 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:26 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:26.826203 25839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:26.817231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:50317)
0619 16:56:26.817828 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:26.817845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:26.818031 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:26.818399 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:26.818405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:26.818903 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:26.819094 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:26.819104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.820216 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.820226 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:26.821190 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:26.821201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.822081 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.822087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:26.822258 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:26.822946 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:26.822965 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:26.825868 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":375,"threads_started":1}
W20260619 16:56:26.826350 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:26.817317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:50317 (local address 127.16.48.62:46429)
0619 16:56:26.817520 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:26.817528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:26.817975 (+ 447us) server_negotiation.cc:407] Connection header received
0619 16:56:26.818187 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:26.818190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:26.818256 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:26.818368 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:26.819255 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.820027 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.821348 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.821877 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.822047 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:26.826104 (+ 4057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:50317: BlockingRecv error: recv got EOF from 127.16.48.3:50317 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 16:56:26 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:26 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:26.862952 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:26.852553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:54565)
0619 16:56:26.853108 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:26.853124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:26.853247 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:26.853676 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:26.853680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:26.854078 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:26.854226 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:26.854234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.855921 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.855926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:26.856963 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:26.856978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.858187 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.858192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:26.858316 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:26.859046 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:26.859065 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:26.862688 (+ 3623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":300,"threads_started":1}
W20260619 16:56:26.863389 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:26.852654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:54565 (local address 127.16.48.62:46429)
0619 16:56:26.852896 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:26.852905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:26.853229 (+ 324us) server_negotiation.cc:407] Connection header received
0619 16:56:26.853373 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:26.853379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:26.853460 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:26.853620 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:26.854403 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.855739 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.857119 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:26.857996 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:26.858160 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:26.863030 (+ 4870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:54565: BlockingRecv error: recv got EOF from 127.16.48.1:54565 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 16:56:27 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:27 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:27.342736 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:27.333104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:37151)
0619 16:56:27.333884 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:27.333901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:27.334095 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:27.334448 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:27.334452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:27.334867 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:27.335014 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:27.335022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.336243 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.336251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:27.337285 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:27.337298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.338121 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.338127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:27.338307 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:27.339059 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:27.339078 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:27.342313 (+ 3235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":521,"threads_started":1}
W20260619 16:56:27.342938 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:27.333183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:37151 (local address 127.16.48.62:46429)
0619 16:56:27.333433 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:27.333443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:27.333995 (+ 552us) server_negotiation.cc:407] Connection header received
0619 16:56:27.334233 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:27.334239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:27.334312 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:27.334424 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:27.335174 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.336044 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.337451 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.337926 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.338094 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:27.342616 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:37151: BlockingRecv error: recv got EOF from 127.16.48.2:37151 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:56:27 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:27 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:27.835721 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:27.827176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:48559)
0619 16:56:27.827793 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:27.827809 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:27.827977 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:27.828349 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:27.828353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:27.828769 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:27.828910 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:27.828919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.829921 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.829927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:27.830967 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:27.830980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.831724 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.831730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:27.831903 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:27.832609 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:27.832627 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:27.835437 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":369,"threads_started":1}
W20260619 16:56:27.835996 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:27.827387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:48559 (local address 127.16.48.62:46429)
0619 16:56:27.827681 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:27.827689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:27.827895 (+ 206us) server_negotiation.cc:407] Connection header received
0619 16:56:27.828099 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:27.828106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:27.828206 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:27.828320 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:27.829069 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.829770 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.831105 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.831551 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.831713 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:27.835735 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:48559: BlockingRecv error: recv got EOF from 127.16.48.3:48559 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Jun 19 16:56:27 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:27 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:27.872445 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:27.864036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:36847)
0619 16:56:27.864581 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:27.864603 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:27.864729 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:27.865044 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:27.865047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:27.865390 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:27.865561 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:27.865569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.866578 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.866583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:27.867510 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:27.867523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.868275 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.868282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:27.868509 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:27.869168 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:27.869185 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:27.872104 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":321,"threads_started":1}
W20260619 16:56:27.872634 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:27.864162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:36847 (local address 127.16.48.62:46429)
0619 16:56:27.864343 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:27.864347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:27.864684 (+ 337us) server_negotiation.cc:407] Connection header received
0619 16:56:27.864835 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:27.864838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:27.864900 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:27.864975 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:27.865663 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.866425 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.867645 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:27.868046 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:27.868230 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:27.872402 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:36847: BlockingRecv error: recv got EOF from 127.16.48.1:36847 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 16:56:28 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:28 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:28.353363 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:28.344046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:50415)
0619 16:56:28.344661 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:28.344678 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:28.344809 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:28.345126 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:28.345129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:28.345476 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:28.345658 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:28.345667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.346787 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.346791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:28.347705 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:28.347717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.348543 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.348547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:28.348697 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:28.349352 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:28.349370 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:28.352982 (+ 3612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":352,"threads_started":1}
W20260619 16:56:28.353478 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:28.344185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:50415 (local address 127.16.48.62:46429)
0619 16:56:28.344444 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:28.344454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:28.344763 (+ 309us) server_negotiation.cc:407] Connection header received
0619 16:56:28.344903 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:28.344906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:28.344972 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:28.345062 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:28.345831 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.346628 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.347903 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.348392 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.348504 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:28.353239 (+ 4735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:50415: BlockingRecv error: recv got EOF from 127.16.48.2:50415 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 16:56:28 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:28 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:28.846521 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:28.836828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:33925)
0619 16:56:28.837466 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:28.837484 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:28.837613 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:28.837926 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:28.837930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:28.838269 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:28.838446 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:28.838454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.839555 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.839560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:28.840709 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:28.840727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.841648 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.841654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:28.841891 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:28.842711 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:28.842738 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:28.846137 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":423,"threads_started":1}
W20260619 16:56:28.846614 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:28.836915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:33925 (local address 127.16.48.62:46429)
0619 16:56:28.837151 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:28.837160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:28.837574 (+ 414us) server_negotiation.cc:407] Connection header received
0619 16:56:28.837724 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:28.837728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:28.837792 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:28.837878 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:28.838576 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.839310 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.840936 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.841410 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.841597 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:28.846360 (+ 4763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:33925: BlockingRecv error: recv got EOF from 127.16.48.3:33925 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 16:56:28 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:28 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:28.883005 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:28.873524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:56725)
0619 16:56:28.874110 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:28.874127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:28.874305 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:28.874695 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:28.874700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:28.875122 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:28.875271 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:28.875280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.876427 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.876434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:28.877423 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:28.877436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.878174 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.878180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:28.878391 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:28.879118 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:28.879142 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:28.882766 (+ 3624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":299,"threads_started":1}
W20260619 16:56:28.883198 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:28.873769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:56725 (local address 127.16.48.62:46429)
0619 16:56:28.873999 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:28.874007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:28.874235 (+ 228us) server_negotiation.cc:407] Connection header received
0619 16:56:28.874457 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:28.874462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:28.874544 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:28.874659 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:28.875431 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.876290 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.877597 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:28.878011 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:28.878158 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:28.882963 (+ 4805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:56725: BlockingRecv error: recv got EOF from 127.16.48.1:56725 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 16:56:29 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:29 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:29.363400 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:29.354677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:33925)
0619 16:56:29.355199 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:29.355216 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:29.355384 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:29.355762 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:29.355766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:29.356230 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:29.356376 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:29.356385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.357629 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.357635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:29.358612 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:29.358626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.359345 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.359351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:29.359537 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:29.360183 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:29.360200 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:29.363033 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":282,"threads_started":1}
W20260619 16:56:29.363516 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:29.354764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:33925 (local address 127.16.48.62:46429)
0619 16:56:29.355041 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:29.355048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:29.355316 (+ 268us) server_negotiation.cc:407] Connection header received
0619 16:56:29.355517 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:29.355523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:29.355596 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:29.355732 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:29.356528 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.357475 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.358770 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.359186 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.359328 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:29.363261 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:33925: BlockingRecv error: recv got EOF from 127.16.48.2:33925 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 19 16:56:29 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:29 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:29.856978 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:29.847664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:34479)
0619 16:56:29.848234 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:29.848252 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:29.848420 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:29.848828 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:29.848833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:29.849254 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:29.849412 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:29.849421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.850590 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.850594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:29.851809 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:29.851823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.852820 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.852825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:29.852966 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:29.853705 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:29.853724 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:29.856736 (+ 3012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":288,"threads_started":1}
W20260619 16:56:29.857205 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:29.847762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:34479 (local address 127.16.48.62:46429)
0619 16:56:29.847991 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:29.847997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:29.848378 (+ 381us) server_negotiation.cc:407] Connection header received
0619 16:56:29.848581 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:29.848587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:29.848669 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:29.848776 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:29.849636 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.850444 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.851985 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.852664 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.852773 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:29.856967 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:34479: BlockingRecv error: recv got EOF from 127.16.48.3:34479 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 16:56:29 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:29 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:29.893298 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:29.884036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:33821)
0619 16:56:29.884645 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:29.884667 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:29.884811 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:29.885213 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:29.885221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:29.885696 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:29.885859 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:29.885869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.886928 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.886934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:29.887893 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:29.887906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.888743 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.888749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:29.888944 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:29.889581 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:29.889603 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:29.892892 (+ 3289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":274,"threads_started":1}
W20260619 16:56:29.893441 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:29.884256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:33821 (local address 127.16.48.62:46429)
0619 16:56:29.884475 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:29.884483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:29.884765 (+ 282us) server_negotiation.cc:407] Connection header received
0619 16:56:29.884959 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:29.884966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:29.885045 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:29.885172 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:29.886012 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.886774 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.888056 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:29.888565 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:29.888725 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:29.893159 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:33821: BlockingRecv error: recv got EOF from 127.16.48.1:33821 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 16:56:30 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:30 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:30.373853 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:30.364539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:48849)
0619 16:56:30.365632 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:30.365656 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:30.365808 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:30.366150 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:30.366158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:30.366681 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:30.366853 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:30.366863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.367893 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.367897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:30.368980 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:30.368996 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.369700 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.369705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:30.369842 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:30.370579 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:30.370599 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:30.373601 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":777,"threads_started":1}
W20260619 16:56:30.374131 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:30.364707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:48849 (local address 127.16.48.62:46429)
0619 16:56:30.364933 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:30.364941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:30.365756 (+ 815us) server_negotiation.cc:407] Connection header received
0619 16:56:30.365927 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:30.365931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:30.365994 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:30.366085 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:30.366997 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.367720 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.369132 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.369533 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.369661 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:30.373860 (+ 4199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:48849: BlockingRecv error: recv got EOF from 127.16.48.2:48849 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 16:56:30 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:30 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:30.867133 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:30.858119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:42875)
0619 16:56:30.858705 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:30.858722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:30.858842 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:30.859163 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:30.859166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:30.859585 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:30.859782 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:30.859792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.860941 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.860946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:30.861874 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:30.861887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.862651 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.862656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:30.862786 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:30.863556 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:30.863584 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:30.866880 (+ 3296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":313,"threads_started":1}
W20260619 16:56:30.867338 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:30.858255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:42875 (local address 127.16.48.62:46429)
0619 16:56:30.858439 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:30.858444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:30.858805 (+ 361us) server_negotiation.cc:407] Connection header received
0619 16:56:30.858949 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:30.858953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:30.859017 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:30.859120 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:30.859926 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.860790 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.862028 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.862474 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.862579 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:30.867075 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:42875: BlockingRecv error: recv got EOF from 127.16.48.3:42875 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 16:56:30 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:30 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:30.902984 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:30.894446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:37465)
0619 16:56:30.894954 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:30.894971 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:30.895089 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:30.895405 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:30.895408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:30.895792 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:30.895943 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:30.895952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.897011 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.897015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:30.897954 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:30.897966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.898647 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.898651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:30.898770 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:30.899457 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:30.899475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:30.902761 (+ 3286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":302,"threads_started":1}
W20260619 16:56:30.903184 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:30.894531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:37465 (local address 127.16.48.62:46429)
0619 16:56:30.894773 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:30.894779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:30.895054 (+ 275us) server_negotiation.cc:407] Connection header received
0619 16:56:30.895201 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:30.895204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:30.895269 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:30.895356 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:30.896088 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.896872 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.898077 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:30.898481 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:30.898576 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:30.902962 (+ 4386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:37465: BlockingRecv error: recv got EOF from 127.16.48.1:37465 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:56:31 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:31 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:31.388495 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:31.375134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:35087)
0619 16:56:31.375770 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:31.375787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:31.375968 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:31.376386 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:31.376391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:31.376832 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:31.376994 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:31.377003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.378059 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.378066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:31.378987 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:31.379000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.379759 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.379766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:31.379946 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:31.384888 (+ 4942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:31.384914 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:31.388083 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":323,"threads_started":1}
W20260619 16:56:31.388772 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:31.375340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:35087 (local address 127.16.48.62:46429)
0619 16:56:31.375654 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:31.375662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:31.375890 (+ 228us) server_negotiation.cc:407] Connection header received
0619 16:56:31.376166 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:31.376171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:31.376236 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:31.376352 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:31.377174 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.377896 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.379173 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.379591 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.379735 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:31.388468 (+ 8733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:35087: BlockingRecv error: recv got EOF from 127.16.48.2:35087 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Jun 19 16:56:31 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:31 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:31.877516 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:31.868494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:34143)
0619 16:56:31.869016 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:31.869032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:31.869169 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:31.869562 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:31.869566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:31.869937 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:31.870116 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:31.870125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.871521 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.871527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:31.872564 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:31.872580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.873394 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.873401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:31.873578 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:31.874266 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:31.874284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:31.877183 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":309,"threads_started":1}
W20260619 16:56:31.877663 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:31.868594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:34143 (local address 127.16.48.62:46429)
0619 16:56:31.868847 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:31.868855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:31.869128 (+ 273us) server_negotiation.cc:407] Connection header received
0619 16:56:31.869316 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:31.869322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:31.869405 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:31.869501 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:31.870255 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.871375 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.872736 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.873203 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.873359 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:31.877404 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:34143: BlockingRecv error: recv got EOF from 127.16.48.3:34143 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 16:56:31 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:31 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:31.913187 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:31.903971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:55225)
0619 16:56:31.904725 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:31.904744 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:31.904870 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:31.905201 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:31.905205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:31.905681 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:31.905849 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:31.905859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.906969 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.906975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:31.908418 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:31.908438 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.909277 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.909281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:31.909398 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:31.909967 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:31.909985 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:31.912920 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":321,"threads_started":1}
W20260619 16:56:31.913403 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:31.904092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:55225 (local address 127.16.48.62:46429)
0619 16:56:31.904300 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:31.904304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:31.904831 (+ 527us) server_negotiation.cc:407] Connection header received
0619 16:56:31.904982 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:31.904986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:31.905052 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:31.905165 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:31.906039 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.906799 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.908560 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:31.909135 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:31.909231 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:31.913152 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:55225: BlockingRecv error: recv got EOF from 127.16.48.1:55225 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 16:56:32 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:32 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:32.398253 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:32.389468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:43161)
0619 16:56:32.390065 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:32.390084 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:32.390213 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:32.390549 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:32.390553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:32.391003 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:32.391187 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:32.391196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.392319 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.392323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:32.393244 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:32.393256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.393941 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.393945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:32.394058 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:32.394777 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:32.394795 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:32.397909 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":285,"threads_started":1}
W20260619 16:56:32.398423 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:32.389631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:43161 (local address 127.16.48.62:46429)
0619 16:56:32.389890 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:32.389897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:32.390172 (+ 275us) server_negotiation.cc:407] Connection header received
0619 16:56:32.390321 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:32.390325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:32.390389 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:32.390473 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:32.391337 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.392180 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.393389 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.393799 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.393899 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:32.398154 (+ 4255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:43161: BlockingRecv error: recv got EOF from 127.16.48.2:43161 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 16:56:32 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:32 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:32.887125 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:32.878504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:51341)
0619 16:56:32.879068 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:32.879085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:32.879209 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:32.879507 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:32.879510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:32.879925 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:32.880092 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:32.880101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.881190 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.881194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:32.882136 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:32.882148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.882930 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.882935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:32.883061 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:32.883663 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:32.883681 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:32.886858 (+ 3177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":293,"threads_started":1}
W20260619 16:56:32.887413 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:32.878567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:51341 (local address 127.16.48.62:46429)
0619 16:56:32.878796 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:32.878806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:32.879176 (+ 370us) server_negotiation.cc:407] Connection header received
0619 16:56:32.879314 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:32.879318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:32.879378 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:32.879471 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:32.880304 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.881041 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.882289 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.882715 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.882824 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:32.887104 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:51341: BlockingRecv error: recv got EOF from 127.16.48.3:51341 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 16:56:32 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:32 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:32.922770 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:32.914137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:55589)
0619 16:56:32.914683 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:32.914704 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:32.914821 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:32.915128 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:32.915131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:32.915485 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:32.915643 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:32.915651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.916716 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.916719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:32.917635 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:32.917647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.918444 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.918448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:32.918607 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:32.919313 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:32.919331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:32.922432 (+ 3101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":255,"threads_started":1}
W20260619 16:56:32.922909 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:32.914306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:55589 (local address 127.16.48.62:46429)
0619 16:56:32.914496 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:32.914502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:32.914790 (+ 288us) server_negotiation.cc:407] Connection header received
0619 16:56:32.914923 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:32.914926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:32.914989 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:32.915073 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:32.915769 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.916583 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.917768 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:32.918280 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:32.918385 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:32.922678 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:55589: BlockingRecv error: recv got EOF from 127.16.48.1:55589 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 16:56:33 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:33 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:33.414281 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:33.399374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:35337)
0619 16:56:33.400001 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:33.400025 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:33.400225 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:33.400571 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:33.400574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:33.400938 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:33.401105 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:33.401114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.402596 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.402602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:33.403538 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:33.403551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.404439 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.404446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:33.404612 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:33.410763 (+ 6151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:33.410788 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:33.413966 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":349,"threads_started":1}
W20260619 16:56:33.414455 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:33.399583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:35337 (local address 127.16.48.62:46429)
0619 16:56:33.399790 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:33.399794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:33.400167 (+ 373us) server_negotiation.cc:407] Connection header received
0619 16:56:33.400366 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:33.400370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:33.400431 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:33.400526 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:33.401265 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.402450 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.403703 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.404199 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.404353 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:33.414195 (+ 9842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:35337: BlockingRecv error: recv got EOF from 127.16.48.2:35337 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 16:56:33 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:33 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:33.896752 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:33.888327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:43081)
0619 16:56:33.888782 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:33.888798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:33.888922 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:33.889240 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:33.889243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:33.889603 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:33.889751 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:33.889759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.890749 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.890760 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:33.891821 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:33.891835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.892683 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.892689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:33.892805 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:33.893475 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:33.893499 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:33.896507 (+ 3008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":230,"threads_started":1}
W20260619 16:56:33.897037 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:33.888394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:43081 (local address 127.16.48.62:46429)
0619 16:56:33.888645 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:33.888651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:33.888887 (+ 236us) server_negotiation.cc:407] Connection header received
0619 16:56:33.889038 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:33.889042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:33.889105 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:33.889192 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:33.889884 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.890579 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.891960 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.892460 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.892561 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:33.896771 (+ 4210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:43081: BlockingRecv error: recv got EOF from 127.16.48.3:43081 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 16:56:33 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:33 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:33.932896 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:33.923779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:37495)
0619 16:56:33.924372 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:33.924390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:33.924516 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:33.924822 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:33.924825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:33.925221 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:33.925385 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:33.925393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.926388 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.926393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:33.927353 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:33.927366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.928084 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.928088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:33.928247 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:33.928859 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:33.928877 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:33.932638 (+ 3761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":317,"threads_started":1}
W20260619 16:56:33.933089 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:33.923860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:37495 (local address 127.16.48.62:46429)
0619 16:56:33.924135 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:33.924143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:33.924501 (+ 358us) server_negotiation.cc:407] Connection header received
0619 16:56:33.924626 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:33.924631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:33.924692 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:33.924779 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:33.925537 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.926249 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.927510 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:33.927926 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:33.928036 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:33.932854 (+ 4818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:37495: BlockingRecv error: recv got EOF from 127.16.48.1:37495 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 16:56:34 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:34 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:34.425158 25862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:34.415317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:42693)
0619 16:56:34.415886 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:34.415904 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:34.416031 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:34.416434 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:34.416437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:34.416880 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:34.417056 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:34.417065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.418147 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.418152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:34.419479 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:34.419495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.420695 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.420700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:34.420880 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:34.421592 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:34.421620 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:34.424868 (+ 3248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":323,"threads_started":1}
W20260619 16:56:34.425467 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:34.415468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:42693 (local address 127.16.48.62:46429)
0619 16:56:34.415661 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:34.415666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:34.415988 (+ 322us) server_negotiation.cc:407] Connection header received
0619 16:56:34.416202 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:34.416206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:34.416271 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:34.416369 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:34.417216 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.417965 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.419659 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.420315 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.420476 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:34.425113 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:42693: BlockingRecv error: recv got EOF from 127.16.48.2:42693 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 16:56:34 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:34 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:34.906567 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:34.897700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:33143)
0619 16:56:34.898268 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:34.898285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:34.898448 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:34.898803 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:34.898808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:34.899253 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:34.899428 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:34.899437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.900593 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.900600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:34.901553 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:34.901565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.902279 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.902285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:34.902475 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:34.903126 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:34.903144 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:34.906186 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":316,"threads_started":1}
W20260619 16:56:34.906678 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:34.897834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:33143 (local address 127.16.48.62:46429)
0619 16:56:34.898077 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:34.898083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:34.898376 (+ 293us) server_negotiation.cc:407] Connection header received
0619 16:56:34.898598 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:34.898603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:34.898665 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:34.898774 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:34.899592 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.900443 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.901702 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.902117 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.902260 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:34.906422 (+ 4162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:33143: BlockingRecv error: recv got EOF from 127.16.48.3:33143 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 16:56:34 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:34 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:34.943151 25864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:34.934032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:59083)
0619 16:56:34.934593 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:34.934610 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:34.934771 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:34.935116 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:34.935119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:34.935522 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:34.935717 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:34.935726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.937013 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.937018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:34.938039 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:34.938053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.938960 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.938965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:34.939094 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:34.939801 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:34.939819 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:34.942807 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":285,"threads_started":1}
W20260619 16:56:34.943279 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:34.934126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:59083 (local address 127.16.48.62:46429)
0619 16:56:34.934337 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:34.934345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:34.934701 (+ 356us) server_negotiation.cc:407] Connection header received
0619 16:56:34.934924 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:34.934928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:34.934992 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:34.935076 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:34.935882 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.936859 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.938204 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:34.938747 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:34.938870 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:34.943054 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:59083: BlockingRecv error: recv got EOF from 127.16.48.1:59083 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 16:56:35 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:35 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:35.435007 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:35.426221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:34863)
0619 16:56:35.426780 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:35.426797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:35.426972 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:35.427343 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:35.427347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:35.427727 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:35.427868 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:35.427876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.429002 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.429008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:35.429941 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:35.429952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.430731 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.430736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:35.430911 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:35.431525 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:35.431551 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:35.434779 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":300,"threads_started":1}
W20260619 16:56:35.435243 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:35.426324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:34863 (local address 127.16.48.62:46429)
0619 16:56:35.426583 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:35.426590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:35.426904 (+ 314us) server_negotiation.cc:407] Connection header received
0619 16:56:35.427121 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:35.427126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:35.427196 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:35.427310 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:35.428023 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.428850 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.430102 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.430520 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.430713 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:35.434981 (+ 4268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:34863: BlockingRecv error: recv got EOF from 127.16.48.2:34863 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 16:56:35 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:35 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:35.916676 25866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:35.907814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:39389)
0619 16:56:35.908373 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:35.908391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:35.908630 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:35.908974 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:35.908980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:35.909492 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:35.909722 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:35.909738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.910768 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.910772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:35.911900 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:35.911914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.912660 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.912664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:35.912783 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:35.913473 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:35.913492 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:35.916425 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":328,"threads_started":1}
W20260619 16:56:35.917022 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:35.908083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:39389 (local address 127.16.48.62:46429)
0619 16:56:35.908309 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:35.908317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:35.908491 (+ 174us) server_negotiation.cc:407] Connection header received
0619 16:56:35.908753 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:35.908759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:35.908832 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:35.908916 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:35.909872 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.910623 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.912036 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.912519 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.912625 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:35.916701 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:39389: BlockingRecv error: recv got EOF from 127.16.48.3:39389 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 19 16:56:35 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:35 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:35.953248 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:35.944449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:58203)
0619 16:56:35.945017 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:35.945039 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:35.945204 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:35.945593 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:35.945599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:35.946213 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:35.946450 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:35.946466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.947522 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.947527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:35.948552 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:35.948566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.949326 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.949331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:35.949462 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:35.950169 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:35.950187 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:35.953004 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":324,"threads_started":1}
W20260619 16:56:35.953441 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:35.944547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:58203 (local address 127.16.48.62:46429)
0619 16:56:35.944747 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:35.944754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:35.945115 (+ 361us) server_negotiation.cc:407] Connection header received
0619 16:56:35.945351 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:35.945356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:35.945434 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:35.945555 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:35.946635 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.947400 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.948683 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:35.949102 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:35.949213 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:35.953209 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:58203: BlockingRecv error: recv got EOF from 127.16.48.1:58203 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 16:56:36 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:36 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:36.450356 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:36.436096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:46349)
0619 16:56:36.436720 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:36.436739 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:36.436868 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:36.437277 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:36.437281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:36.437696 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:36.437894 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:36.437903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.438933 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.438938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:36.439853 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:36.439865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.440608 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.440614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:36.440742 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:36.446997 (+ 6255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:36.447020 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:36.450100 (+ 3080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":324,"threads_started":1}
W20260619 16:56:36.450608 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:36.436252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:46349 (local address 127.16.48.62:46429)
0619 16:56:36.436520 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:36.436526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:36.436856 (+ 330us) server_negotiation.cc:407] Connection header received
0619 16:56:36.437057 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:36.437062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:36.437127 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:36.437220 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:36.438060 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.438789 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.439997 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.440442 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.440535 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:36.450325 (+ 9790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:46349: BlockingRecv error: recv got EOF from 127.16.48.2:46349 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 16:56:36 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:36 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:36.927387 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:36.917732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:50769)
0619 16:56:36.918334 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:36.918352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:36.918597 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:36.919005 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:36.919010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:36.919482 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:36.919680 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:36.919690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.921212 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.921224 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:36.922394 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:36.922412 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.923152 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.923157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:36.923294 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:36.923982 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:36.924002 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:36.927146 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":328,"threads_started":1}
W20260619 16:56:36.927731 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:36.917964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:50769 (local address 127.16.48.62:46429)
0619 16:56:36.918192 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:36.918201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:36.918496 (+ 295us) server_negotiation.cc:407] Connection header received
0619 16:56:36.918760 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:36.918765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:36.918839 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:36.918977 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:36.919851 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.921022 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.922567 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.922998 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.923102 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:36.927390 (+ 4288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:50769: BlockingRecv error: recv got EOF from 127.16.48.3:50769 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 16:56:36 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:36 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:36.963323 25870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:36.954333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:42125)
0619 16:56:36.954894 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:36.954911 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:36.955097 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:36.955470 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:36.955474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:36.955909 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:36.956071 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:36.956080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.957148 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.957155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:36.958501 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:36.958517 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.959398 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.959402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:36.959533 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:36.960248 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:36.960266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:36.963080 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":302,"threads_started":1}
W20260619 16:56:36.963563 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:36.954395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:42125 (local address 127.16.48.62:46429)
0619 16:56:36.954735 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:36.954743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:36.955007 (+ 264us) server_negotiation.cc:407] Connection header received
0619 16:56:36.955244 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:36.955248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:36.955321 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:36.955431 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:36.956282 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.956990 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.958667 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:36.959261 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:36.959359 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:36.963294 (+ 3935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:42125: BlockingRecv error: recv got EOF from 127.16.48.1:42125 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Jun 19 16:56:37 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:37 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:37.460635 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:37.451429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:41897)
0619 16:56:37.452293 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:37.452316 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:37.452447 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:37.452871 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:37.452875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:37.453330 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:37.453488 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:37.453498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.454577 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.454581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:37.455504 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:37.455516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.456458 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.456463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:37.456653 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:37.457311 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:37.457334 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:37.460398 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":610,"threads_started":1}
W20260619 16:56:37.461040 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:37.451533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:41897 (local address 127.16.48.62:46429)
0619 16:56:37.451831 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:37.451841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:37.452410 (+ 569us) server_negotiation.cc:407] Connection header received
0619 16:56:37.452562 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:37.452566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:37.452677 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:37.452776 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:37.453660 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.454402 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.455648 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.456250 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.456378 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:37.460666 (+ 4288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:41897: BlockingRecv error: recv got EOF from 127.16.48.2:41897 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 16:56:37 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:37 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:37.937786 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:37.928565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:42861)
0619 16:56:37.929142 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:37.929158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:37.929283 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:37.929604 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:37.929609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:37.929983 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:37.930160 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:37.930168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.931284 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.931288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:37.932336 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:37.932350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.933092 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.933097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:37.933237 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:37.933967 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:37.933991 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:37.937522 (+ 3531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":309,"threads_started":1}
W20260619 16:56:37.938052 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:37.928619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:42861 (local address 127.16.48.62:46429)
0619 16:56:37.928805 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:37.928814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:37.929246 (+ 432us) server_negotiation.cc:407] Connection header received
0619 16:56:37.929388 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:37.929392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:37.929456 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:37.929554 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:37.930333 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.931146 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.932515 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.932953 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.933066 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:37.937782 (+ 4716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:42861: BlockingRecv error: recv got EOF from 127.16.48.3:42861 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 16:56:37 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:37 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:37.973483 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:37.964478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:46503)
0619 16:56:37.965007 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:37.965023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:37.965203 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:37.965575 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:37.965580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:37.966031 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:37.966193 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:37.966203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.967288 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.967294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:37.968344 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:37.968359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.969127 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.969134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:37.969298 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:37.969912 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:37.969930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:37.973092 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":268,"threads_started":1}
W20260619 16:56:37.973624 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:37.964585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:46503 (local address 127.16.48.62:46429)
0619 16:56:37.964807 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:37.964816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:37.965120 (+ 304us) server_negotiation.cc:407] Connection header received
0619 16:56:37.965340 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:37.965346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:37.965433 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:37.965546 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:37.966348 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.967128 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.968492 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:37.968923 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:37.969061 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:37.973338 (+ 4277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:46503: BlockingRecv error: recv got EOF from 127.16.48.1:46503 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 16:56:38 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:38 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:38.470559 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:38.461780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:38099)
0619 16:56:38.462357 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:38.462384 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:38.462546 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:38.462913 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:38.462917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:38.463351 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:38.463498 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:38.463507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.464615 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.464620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:38.465539 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:38.465550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.466396 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.466402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:38.466572 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:38.467319 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:38.467338 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:38.470193 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":304,"threads_started":1}
W20260619 16:56:38.470676 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:38.462063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:38099 (local address 127.16.48.62:46429)
0619 16:56:38.462322 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:38.462328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:38.462472 (+ 144us) server_negotiation.cc:407] Connection header received
0619 16:56:38.462697 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:38.462702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:38.462773 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:38.462883 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:38.463651 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.464465 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.465693 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.466239 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.466376 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:38.470425 (+ 4049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:38099: BlockingRecv error: recv got EOF from 127.16.48.2:38099 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 16:56:38 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:38 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:38.947717 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:38.938946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:56147)
0619 16:56:38.939490 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:38.939507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:38.939623 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:38.939954 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:38.939958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:38.940443 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:38.940628 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:38.940637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.941696 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.941701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:38.942725 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:38.942739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.943655 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.943665 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:38.943819 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:38.944574 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:38.944592 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:38.947467 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":277,"threads_started":1}
W20260619 16:56:38.947945 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:38.938992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:56147 (local address 127.16.48.62:46429)
0619 16:56:38.939207 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:38.939213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:38.939587 (+ 374us) server_negotiation.cc:407] Connection header received
0619 16:56:38.939717 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:38.939720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:38.939782 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:38.939863 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:38.940776 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.941521 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.942896 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.943423 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.943527 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:38.947687 (+ 4160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:56147: BlockingRecv error: recv got EOF from 127.16.48.3:56147 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 16:56:38 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:38 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:38.983366 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:38.974775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:57105)
0619 16:56:38.975606 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:38.975623 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:38.975772 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:38.976168 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:38.976175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:38.976622 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:38.976795 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:38.976804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.977837 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.977841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:38.978741 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:38.978752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.979472 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.979477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:38.979621 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:38.980325 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:38.980343 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:38.983095 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":398,"threads_started":1}
W20260619 16:56:38.983541 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:38.974949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:57105 (local address 127.16.48.62:46429)
0619 16:56:38.975270 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:38.975278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:38.975727 (+ 449us) server_negotiation.cc:407] Connection header received
0619 16:56:38.975915 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:38.975920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:38.976018 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:38.976101 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:38.976951 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.977710 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.978862 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:38.979294 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:38.979394 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:38.983317 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:57105: BlockingRecv error: recv got EOF from 127.16.48.1:57105 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
Jun 19 16:56:39 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:39 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:39.480824 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:39.471726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:41323)
0619 16:56:39.472321 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:39.472338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:39.472461 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:39.472789 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:39.472792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:39.473125 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:39.473303 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:39.473311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.474333 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.474337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:39.475685 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:39.475701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.476651 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.476655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:39.476790 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:39.477551 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:39.477570 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:39.480584 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":291,"threads_started":1}
W20260619 16:56:39.481158 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:39.471912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:41323 (local address 127.16.48.62:46429)
0619 16:56:39.472210 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:39.472214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:39.472424 (+ 210us) server_negotiation.cc:407] Connection header received
0619 16:56:39.472571 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:39.472574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:39.472645 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:39.472744 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:39.473441 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.474170 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.475864 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.476504 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.476615 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:39.480856 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:41323: BlockingRecv error: recv got EOF from 127.16.48.2:41323 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 16:56:39 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:39 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.3@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:39.957886 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:39.948868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.3:51073)
0619 16:56:39.949334 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:39.949351 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:39.949481 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:39.949786 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:39.949790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:39.950217 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:39.950392 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:39.950402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.951793 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.951797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:39.952800 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:39.952812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.953551 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.953555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:39.953778 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:39.954501 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:39.954525 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:39.957635 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":268,"threads_started":1}
W20260619 16:56:39.958143 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:39.948975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:51073 (local address 127.16.48.62:46429)
0619 16:56:39.949229 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:39.949236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:39.949437 (+ 201us) server_negotiation.cc:407] Connection header received
0619 16:56:39.949588 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:39.949591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:39.949653 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:39.949738 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:39.950820 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.951657 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.952934 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.953394 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.953528 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:39.957871 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.3:51073: BlockingRecv error: recv got EOF from 127.16.48.3:51073 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 19 16:56:39 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:39 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.1@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:39.993413 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:39.984703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.1:59261)
0619 16:56:39.985177 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:39.985194 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:39.985317 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:39.985653 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:39.985656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:39.986186 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:39.986377 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:39.986386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.987417 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.987422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:39.988376 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:39.988388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.989112 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.989117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:39.989242 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:39.989853 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:39.989870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:39.993066 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":280,"threads_started":1}
W20260619 16:56:39.993609 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:39.984845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:59261 (local address 127.16.48.62:46429)
0619 16:56:39.985097 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:39.985103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:39.985280 (+ 177us) server_negotiation.cc:407] Connection header received
0619 16:56:39.985426 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:39.985429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:39.985518 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:39.985607 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:39.986536 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.987259 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.988523 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:39.988926 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:39.989036 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:39.993335 (+ 4299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.1:59261: BlockingRecv error: recv got EOF from 127.16.48.1:59261 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 19 16:56:40 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:56:40 dist-test-slave-3ffd krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.48.2@KRBTEST.COM for oryx/127.16.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:56:40.491271 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:40.482118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46429 (local address 127.16.48.2:39075)
0619 16:56:40.482663 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:40.482680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:40.482824 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:40.483171 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:40.483176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:40.483605 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:40.483791 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:40.483800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.484870 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.484875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:40.485854 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:40.485867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.486635 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.486641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:40.486817 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:40.487581 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:40.487620 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:40.491006 (+ 3386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.48.62:46429: Server oryx/127.16.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":266,"threads_started":1}
W20260619 16:56:40.491555 25835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:40.482193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:39075 (local address 127.16.48.62:46429)
0619 16:56:40.482407 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:40.482414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:40.482759 (+ 345us) server_negotiation.cc:407] Connection header received
0619 16:56:40.482950 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:40.482957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:40.483022 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:40.483134 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:40.483935 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.484722 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.486000 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.486456 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.486603 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:40.491274 (+ 4671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.48.2:39075: BlockingRecv error: recv got EOF from 127.16.48.2:39075 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:56:40.783427 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25420
I20260619 16:56:40.797571 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:40.784858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.16.48.1:39349)
0619 16:56:40.785534 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:40.785542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:40.785571 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:56:40.785635 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:40.785642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:40.785902 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:40.786069 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:40.787612 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.788995 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.790282 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.790886 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.791000 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:40.793794 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:40.793804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:40.793821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:40.793863 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:40.796870 (+ 3007us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.48.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 16:56:40.797258 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57670: Request ticket server kudu/127.16.48.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":342,"threads_started":1}
W20260619 16:56:40.797631 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25420: Not authorized: Client connection negotiation failed: client connection to 127.16.48.1:39349: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.48.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 16:56:40.797780 25882 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57670: Request ticket server kudu/127.16.48.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 16:56:40.797909 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 25420
I20260619 16:56:40.812525 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25572
I20260619 16:56:40.825724 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:40.813784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.16.48.2:37401)
0619 16:56:40.814463 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:40.814472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:40.814508 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:56:40.814570 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:40.814577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:40.814789 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:40.814954 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:40.816314 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.817403 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.818821 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.819442 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.819574 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:40.822344 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:40.822354 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:40.822370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:40.822426 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:40.825178 (+ 2752us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.48.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 16:56:40.825473 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39716: Request ticket server kudu/127.16.48.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":362,"threads_started":1}
W20260619 16:56:40.825871 25883 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39716: Request ticket server kudu/127.16.48.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 16:56:40.825996 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25572: Not authorized: Client connection negotiation failed: client connection to 127.16.48.2:37401: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.48.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 16:56:40.826174 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 25572
I20260619 16:56:40.843982 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25704
I20260619 16:56:40.857350 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:40.845007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.16.48.3:33851)
0619 16:56:40.845660 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:40.845667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:40.845694 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:56:40.845754 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:40.845761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:40.845983 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:40.846138 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:40.847373 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.848509 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.849832 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:40.850533 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:40.850744 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:40.853584 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:40.853599 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:40.853619 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:40.853698 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:40.856629 (+ 2931us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.48.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 16:56:40.857006 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43756: Request ticket server kudu/127.16.48.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":371,"threads_started":1}
W20260619 16:56:40.857465 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25704: Not authorized: Client connection negotiation failed: client connection to 127.16.48.3:33851: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.48.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 16:56:40.857497 25884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43756: Request ticket server kudu/127.16.48.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 16:56:40.857642 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 25704
I20260619 16:56:40.874785 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25351
I20260619 16:56:40.876099 25413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:40.990356 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 25351
2026-06-19T16:56:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21472 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T16:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:56:41Z Disabled control of system clock
I20260619 16:56:41.071094 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46517
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:42687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:46517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:56:41.418133 25889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:41.418519 25889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:41.418658 25889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:41.426297 25889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:41.426479 25889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:41.426577 25889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:41.426683 25889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:41.426771 25889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:41.436415 25889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:42687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:46517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:41.437932 25889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:41.439903 25889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:41.453377 25895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:41.453576 25889 server_base.cc:1034] running on GCE node
W20260619 16:56:41.453403 25894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:41.453647 25897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:41.454748 25889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:41.456434 25889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:41.457665 25889 hybrid_clock.cc:648] HybridClock initialized: now 1781888201457615 us; error 41 us; skew 500 ppm
I20260619 16:56:41.458086 25889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:56:41.466943 25889 webserver.cc:460] Webserver started at http://127.16.48.62:41139/ using document root <none> and password file <none>
I20260619 16:56:41.467855 25889 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:41.467996 25889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:41.468400 25889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:41.471553 25889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "1012249f12d6416aa554924dbd8cf0db"
format_stamp: "Formatted at 2026-06-19 16:56:41 on dist-test-slave-3ffd"
server_key: "da70036e31169be53d92263020ecc45a"
server_key_iv: "de754996782197fc329e38c7d49ffeba"
server_key_version: "encryptionkey@0"
I20260619 16:56:41.472472 25889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "1012249f12d6416aa554924dbd8cf0db"
format_stamp: "Formatted at 2026-06-19 16:56:41 on dist-test-slave-3ffd"
server_key: "da70036e31169be53d92263020ecc45a"
server_key_iv: "de754996782197fc329e38c7d49ffeba"
server_key_version: "encryptionkey@0"
I20260619 16:56:41.478070 25889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:56:41.481793 25903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:41.483415 25889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:56:41.483608 25889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "1012249f12d6416aa554924dbd8cf0db"
format_stamp: "Formatted at 2026-06-19 16:56:41 on dist-test-slave-3ffd"
server_key: "da70036e31169be53d92263020ecc45a"
server_key_iv: "de754996782197fc329e38c7d49ffeba"
server_key_version: "encryptionkey@0"
I20260619 16:56:41.483830 25889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:41.497296 25889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:41.498199 25889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:41.498523 25889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:41.517208 25889 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:46517
I20260619 16:56:41.517215 25954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:46517 every 8 connection(s)
I20260619 16:56:41.518802 25889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:56:41.523010 25955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:41.525830 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 25889
I20260619 16:56:41.526099 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:56:41.526837 16576 external_mini_cluster.cc:1428] Setting key f05a29441b3cb1cf17b80c1a0ac6ee70
I20260619 16:56:41.536583 25955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db: Bootstrap starting.
I20260619 16:56:41.540745 25955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:41.542014 25955 log.cc:824] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:41.542881 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:41.529242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.16.48.62:46517)
0619 16:56:41.530405 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:41.530417 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:41.530472 (+ 55us) server_negotiation.cc:407] Connection header received
0619 16:56:41.532367 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:41.532403 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:41.533010 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:41.533459 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:41.534206 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:41.535676 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:41.537640 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:41.538443 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:41.538615 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:41.539988 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:41.539996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:41.540013 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:41.540703 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:41.540803 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:41.540826 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:41.540977 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:41.541306 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 16:56:41.541656 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":395,"threads_started":1}
I20260619 16:56:41.545365 25955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db: No bootstrap required, opened a new log
I20260619 16:56:41.553694 25955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012249f12d6416aa554924dbd8cf0db" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46517 } }
I20260619 16:56:41.554106 25955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:41.554237 25955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1012249f12d6416aa554924dbd8cf0db, State: Initialized, Role: FOLLOWER
I20260619 16:56:41.554965 25955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012249f12d6416aa554924dbd8cf0db" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46517 } }
I20260619 16:56:41.555234 25955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:56:41.555435 25955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:56:41.555617 25955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:41.557927 25955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012249f12d6416aa554924dbd8cf0db" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46517 } }
I20260619 16:56:41.558511 25955 leader_election.cc:302] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1012249f12d6416aa554924dbd8cf0db; no voters:
I20260619 16:56:41.559150 25955 leader_election.cc:288] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:56:41.559345 25960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:41.560096 25960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [term 1 LEADER]: Becoming Leader. State: Replica: 1012249f12d6416aa554924dbd8cf0db, State: Running, Role: LEADER
I20260619 16:56:41.560761 25960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012249f12d6416aa554924dbd8cf0db" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46517 } }
I20260619 16:56:41.561381 25955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:56:41.564303 25962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1012249f12d6416aa554924dbd8cf0db. Latest consensus state: current_term: 1 leader_uuid: "1012249f12d6416aa554924dbd8cf0db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012249f12d6416aa554924dbd8cf0db" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46517 } } }
I20260619 16:56:41.564235 25961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1012249f12d6416aa554924dbd8cf0db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012249f12d6416aa554924dbd8cf0db" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46517 } } }
I20260619 16:56:41.564738 25961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:41.564740 25962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:41.566128 25968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:56:41.572569 25968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:56:41.582448 25968 catalog_manager.cc:1347] Generated new cluster ID: 61d82f1feef043deb59b37f964a9955e
I20260619 16:56:41.582645 25968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:56:41.597200 25968 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:56:41.598623 25968 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:56:41.606896 25968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1012249f12d6416aa554924dbd8cf0db: Generated new TSK 0
I20260619 16:56:41.607872 25968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:56:41.616195 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46517
--builtin_ntp_servers=127.16.48.20:42687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 16:56:41.986666 25979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:41.986977 25979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:41.987125 25979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:41.994886 25979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:41.995016 25979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:41.995196 25979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:56:42.005631 25979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:42687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:42.007058 25979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:42.009107 25979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:42.023487 25987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:42.023520 25985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:42.023645 25984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:42.024246 25979 server_base.cc:1034] running on GCE node
I20260619 16:56:42.024930 25979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:42.026026 25979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:42.027230 25979 hybrid_clock.cc:648] HybridClock initialized: now 1781888202027162 us; error 76 us; skew 500 ppm
I20260619 16:56:42.027570 25979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:56:42.030093 25979 webserver.cc:460] Webserver started at http://127.16.48.1:37549/ using document root <none> and password file <none>
I20260619 16:56:42.030866 25979 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:42.030962 25979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:42.031291 25979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:42.034298 25979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "035ed46fd1384f3da0e99deef9be1833"
format_stamp: "Formatted at 2026-06-19 16:56:42 on dist-test-slave-3ffd"
server_key: "9454c0a34aceec2f8dccb087d1f4e2db"
server_key_iv: "47ee9c338028a26b4c4ab59d64f722be"
server_key_version: "encryptionkey@0"
I20260619 16:56:42.035120 25979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "035ed46fd1384f3da0e99deef9be1833"
format_stamp: "Formatted at 2026-06-19 16:56:42 on dist-test-slave-3ffd"
server_key: "9454c0a34aceec2f8dccb087d1f4e2db"
server_key_iv: "47ee9c338028a26b4c4ab59d64f722be"
server_key_version: "encryptionkey@0"
I20260619 16:56:42.040844 25979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:56:42.044872 25993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:42.046536 25979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:42.046761 25979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "035ed46fd1384f3da0e99deef9be1833"
format_stamp: "Formatted at 2026-06-19 16:56:42 on dist-test-slave-3ffd"
server_key: "9454c0a34aceec2f8dccb087d1f4e2db"
server_key_iv: "47ee9c338028a26b4c4ab59d64f722be"
server_key_version: "encryptionkey@0"
I20260619 16:56:42.047022 25979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:42.072795 25979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:42.073853 25979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:42.074159 25979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:42.075207 25979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:42.076798 25979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:42.076923 25979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:42.077056 25979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:42.077142 25979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:42.111940 25979 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:33507
I20260619 16:56:42.112052 26105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:33507 every 8 connection(s)
I20260619 16:56:42.113760 25979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:56:42.117583 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 25979
I20260619 16:56:42.117785 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:56:42.118641 16576 external_mini_cluster.cc:1428] Setting key be7eea8960e4c605a7e69aadfbdec8f1
I20260619 16:56:42.121981 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46517
--builtin_ntp_servers=127.16.48.20:42687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:56:42.131031 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:42.116879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:58925 (local address 127.16.48.62:46517)
0619 16:56:42.117389 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:42.117395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:42.119081 (+ 1686us) server_negotiation.cc:407] Connection header received
0619 16:56:42.121590 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:42.121595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:42.121662 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:42.121756 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:42.123495 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:42.124367 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:42.127193 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:42.127632 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:42.127758 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:42.129806 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:42.129812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:42.129816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:42.129837 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:42.129927 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:42.129950 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:42.130089 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:42.130453 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 16:56:42.130590 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":257,"threads_started":1}
I20260619 16:56:42.132187 26109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:42.117574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46517 (local address 127.16.48.1:58925)
0619 16:56:42.118874 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:42.118928 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:42.121228 (+ 2300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:42.122094 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:42.122110 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:42.122272 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:42.123237 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:42.123271 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:42.124574 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:42.124583 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:42.126917 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:42.126943 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:42.127931 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:42.127941 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:42.128364 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:42.129273 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:56:42.129279 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:42.129348 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:56:42.129495 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:42.130181 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:42.130191 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:42.130199 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:56:42.130398 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 16:56:42.130822 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":347,"threads_started":1}
I20260619 16:56:42.134037 26106 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46517
I20260619 16:56:42.134418 26106 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:42.135458 26106 heartbeater.cc:511] Master 127.16.48.62:46517 requested a full tablet report, sending...
I20260619 16:56:42.137900 25920 ts_manager.cc:194] Registered new tserver with Master: 035ed46fd1384f3da0e99deef9be1833 (127.16.48.1:33507)
I20260619 16:56:42.139479 25920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.1:58925
W20260619 16:56:42.471335 26110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:42.471649 26110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:42.471736 26110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:42.479208 26110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:42.479331 26110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:42.479439 26110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:56:42.489156 26110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:42687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:42.490481 26110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:42.492468 26110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:42.507194 26118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:42.507288 26116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:42.507299 26115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:42.513679 26110 server_base.cc:1034] running on GCE node
I20260619 16:56:42.514410 26110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:42.515512 26110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:42.516769 26110 hybrid_clock.cc:648] HybridClock initialized: now 1781888202516725 us; error 45 us; skew 500 ppm
I20260619 16:56:42.517226 26110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:56:42.520452 26110 webserver.cc:460] Webserver started at http://127.16.48.2:34579/ using document root <none> and password file <none>
I20260619 16:56:42.521476 26110 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:42.521615 26110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:42.522027 26110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:42.526436 26110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "f60c030cba5b42f8907e271901dab45a"
format_stamp: "Formatted at 2026-06-19 16:56:42 on dist-test-slave-3ffd"
server_key: "673d8dc43326d032907e8e814ad08c63"
server_key_iv: "f13535c7c3eb356b42e3f76f2b89fa4e"
server_key_version: "encryptionkey@0"
I20260619 16:56:42.527673 26110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "f60c030cba5b42f8907e271901dab45a"
format_stamp: "Formatted at 2026-06-19 16:56:42 on dist-test-slave-3ffd"
server_key: "673d8dc43326d032907e8e814ad08c63"
server_key_iv: "f13535c7c3eb356b42e3f76f2b89fa4e"
server_key_version: "encryptionkey@0"
I20260619 16:56:42.535460 26110 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260619 16:56:42.540634 26124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:42.542354 26110 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:56:42.542608 26110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "f60c030cba5b42f8907e271901dab45a"
format_stamp: "Formatted at 2026-06-19 16:56:42 on dist-test-slave-3ffd"
server_key: "673d8dc43326d032907e8e814ad08c63"
server_key_iv: "f13535c7c3eb356b42e3f76f2b89fa4e"
server_key_version: "encryptionkey@0"
I20260619 16:56:42.542876 26110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:42.556576 26110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:42.557533 26110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:42.557863 26110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:42.558969 26110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:42.560681 26110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:42.560814 26110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:42.560956 26110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:42.561050 26110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:42.596426 26110 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:36313
I20260619 16:56:42.596519 26236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:36313 every 8 connection(s)
I20260619 16:56:42.597872 26110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:56:42.600708 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 26110
I20260619 16:56:42.600886 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:56:42.601447 16576 external_mini_cluster.cc:1428] Setting key 4d17a7ee190cfa18ba54a4ab60faa649
I20260619 16:56:42.604826 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46517
--builtin_ntp_servers=127.16.48.20:42687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:56:42.613796 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:42.600731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:55657 (local address 127.16.48.62:46517)
0619 16:56:42.601142 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:42.601149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:42.602771 (+ 1622us) server_negotiation.cc:407] Connection header received
0619 16:56:42.604972 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:42.604978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:42.605084 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:42.605222 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:42.606811 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:42.607584 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:42.610271 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:42.610698 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:42.610812 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:42.612698 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:42.612704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:42.612708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:42.612731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:42.612768 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:42.612782 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:42.612934 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:42.613230 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 16:56:42.613346 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260619 16:56:42.615329 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:42.601332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46517 (local address 127.16.48.2:55657)
0619 16:56:42.602493 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:42.602546 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:42.604597 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:42.605531 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:42.605547 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:42.605694 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:42.606599 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:42.606629 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:42.607745 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:42.607754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:42.610058 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:42.610080 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:42.610963 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:42.610972 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:42.611347 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:42.612226 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:56:42.612231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:42.612298 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:56:42.612422 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:42.612937 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:42.612952 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:42.612964 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 16:56:42.613204 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 16:56:42.613701 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":352,"threads_started":1}
I20260619 16:56:42.617090 26237 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46517
I20260619 16:56:42.617447 26237 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:42.618389 26237 heartbeater.cc:511] Master 127.16.48.62:46517 requested a full tablet report, sending...
I20260619 16:56:42.619984 25920 ts_manager.cc:194] Registered new tserver with Master: f60c030cba5b42f8907e271901dab45a (127.16.48.2:36313)
I20260619 16:56:42.620847 25920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.2:55657
W20260619 16:56:42.961043 26240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:42.961404 26240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:42.961771 26240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:42.969811 26240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:42.969993 26240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:42.970187 26240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:56:42.979873 26240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:42687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:46517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:42.981395 26240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:42.983397 26240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:42.998672 26245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:42.998812 26248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:42.998656 26246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:42.999732 26240 server_base.cc:1034] running on GCE node
I20260619 16:56:43.000521 26240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:43.001663 26240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:43.002928 26240 hybrid_clock.cc:648] HybridClock initialized: now 1781888203002876 us; error 85 us; skew 500 ppm
I20260619 16:56:43.003278 26240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:56:43.006070 26240 webserver.cc:460] Webserver started at http://127.16.48.3:39499/ using document root <none> and password file <none>
I20260619 16:56:43.006853 26240 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:43.006994 26240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:43.007334 26240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:43.010505 26240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "277db226e05f40cbbfea2e64905a365d"
format_stamp: "Formatted at 2026-06-19 16:56:43 on dist-test-slave-3ffd"
server_key: "96c9cdb58b46b6dd94def22fd057cb77"
server_key_iv: "7db1979a2d36334bc91627154b623d02"
server_key_version: "encryptionkey@0"
I20260619 16:56:43.011385 26240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "277db226e05f40cbbfea2e64905a365d"
format_stamp: "Formatted at 2026-06-19 16:56:43 on dist-test-slave-3ffd"
server_key: "96c9cdb58b46b6dd94def22fd057cb77"
server_key_iv: "7db1979a2d36334bc91627154b623d02"
server_key_version: "encryptionkey@0"
I20260619 16:56:43.017346 26240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 16:56:43.021330 26254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:43.022955 26240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:56:43.023176 26240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "277db226e05f40cbbfea2e64905a365d"
format_stamp: "Formatted at 2026-06-19 16:56:43 on dist-test-slave-3ffd"
server_key: "96c9cdb58b46b6dd94def22fd057cb77"
server_key_iv: "7db1979a2d36334bc91627154b623d02"
server_key_version: "encryptionkey@0"
I20260619 16:56:43.023419 26240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:43.048388 26240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:43.049326 26240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:43.049654 26240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:43.050786 26240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:43.052608 26240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:43.052722 26240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:43.052855 26240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:43.052940 26240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:43.089469 26240 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:46109
I20260619 16:56:43.089573 26366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:46109 every 8 connection(s)
I20260619 16:56:43.090878 26240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:56:43.093686 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 26240
I20260619 16:56:43.093868 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:56:43.094337 16576 external_mini_cluster.cc:1428] Setting key bce3e79fa16c9cf7bef4d805fa7de15d
I20260619 16:56:43.104832 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:43.093805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:33175 (local address 127.16.48.62:46517)
0619 16:56:43.094020 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:43.094027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:43.095902 (+ 1875us) server_negotiation.cc:407] Connection header received
0619 16:56:43.097765 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:43.097769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:43.097838 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:43.097926 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:43.099063 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.099827 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.101635 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.102124 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.102240 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:43.104061 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:43.104067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:43.104070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:43.104092 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:43.104190 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:43.104207 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:43.104297 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:43.104496 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 16:56:43.104580 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:56:43.106007 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:43.094496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:46517 (local address 127.16.48.3:33175)
0619 16:56:43.095716 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:43.095776 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:43.097489 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:43.098146 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:43.098155 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:43.098263 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:43.098894 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:43.098911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.100009 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.100013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:43.101439 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:43.101453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.102372 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.102378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:43.102626 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:43.103647 (+ 1021us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:56:43.103652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:43.103725 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:56:43.103858 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:43.104296 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:43.104303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:43.104310 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:56:43.104478 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 16:56:43.104826 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":389,"threads_started":1}
I20260619 16:56:43.107722 26367 heartbeater.cc:347] Connected to a master server at 127.16.48.62:46517
I20260619 16:56:43.108109 26367 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:43.109068 26367 heartbeater.cc:511] Master 127.16.48.62:46517 requested a full tablet report, sending...
I20260619 16:56:43.110632 25920 ts_manager.cc:194] Registered new tserver with Master: 277db226e05f40cbbfea2e64905a365d (127.16.48.3:46109)
I20260619 16:56:43.111750 25920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.3:33175
I20260619 16:56:43.122992 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:56:43.129657 26108 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:43.128076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.16.48.62:46517)
0619 16:56:43.128313 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:43.128317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:43.128782 (+ 465us) server_negotiation.cc:407] Connection header received
0619 16:56:43.128921 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:43.128925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:43.129003 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:43.129110 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:43.129461 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57466: BlockingRecv error: recv got EOF from 127.0.0.1:57466 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 16:56:43.130568 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25979
I20260619 16:56:43.139544 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:43.131738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.16.48.1:33507)
0619 16:56:43.132491 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:43.132508 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:43.132537 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:56:43.132598 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:43.132604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:43.132837 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:43.132978 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:43.133436 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.134463 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.135853 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.136613 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.136737 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:43.137886 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:43.137895 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:43.137910 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:43.138589 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:43.138640 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:43.138662 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:43.138765 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:43.138976 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 16:56:43.139150 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":493,"threads_started":1}
I20260619 16:56:43.141069 26101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:43.143986 26106 heartbeater.cc:503] Master 127.16.48.62:46517 was elected leader, sending a full tablet report...
I20260619 16:56:43.225605 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 25979
W20260619 16:56:43.242336 25907 connection.cc:535] server connection from 127.16.48.1:58925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:56:43.242713 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 26110
I20260619 16:56:43.251044 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:43.243574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.16.48.2:36313)
0619 16:56:43.244109 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:43.244156 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:43.244194 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:56:43.244254 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:43.244260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:43.244498 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:43.244648 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:43.245032 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.246038 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.247296 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.248041 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.248287 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:43.249456 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:43.249464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:43.249478 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:43.250129 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:43.250180 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:43.250200 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:43.250361 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:43.250603 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 16:56:43.250740 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":273,"threads_started":1}
I20260619 16:56:43.252405 26232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:43.335635 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 26110
I20260619 16:56:43.352237 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 26240
I20260619 16:56:43.360890 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:43.353224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.16.48.3:46109)
0619 16:56:43.353811 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:43.353818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:43.353848 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:56:43.353910 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:43.353917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:43.354152 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:43.354340 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:43.354813 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.355859 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.357206 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:43.357878 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:43.358023 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:43.359257 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:43.359267 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:43.359283 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:43.359954 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:43.360004 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:43.360025 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:43.360323 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:43.360395 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 16:56:43.360566 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":284,"threads_started":1}
I20260619 16:56:43.362675 26362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:43.446916 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 26240
I20260619 16:56:43.464442 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 25889
I20260619 16:56:43.465704 25950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:43.580106 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 25889
2026-06-19T16:56:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2585 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T16:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:56:43Z Disabled control of system clock
I20260619 16:56:43.656636 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:44689
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:44689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:56:44.002585 26386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:44.003003 26386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:44.003175 26386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:44.010802 26386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:44.010988 26386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:44.011090 26386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:44.011198 26386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:44.011289 26386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:44.020813 26386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:44689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:44689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:44.022262 26386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:44.024303 26386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:44.037952 26394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:44.038002 26392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:44.037927 26391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:44.038852 26386 server_base.cc:1034] running on GCE node
I20260619 16:56:44.039724 26386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:44.041299 26386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:44.042508 26386 hybrid_clock.cc:648] HybridClock initialized: now 1781888204042456 us; error 37 us; skew 500 ppm
I20260619 16:56:44.042920 26386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:56:44.045467 26386 webserver.cc:460] Webserver started at http://127.16.48.62:42791/ using document root <none> and password file <none>
I20260619 16:56:44.046224 26386 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:44.046353 26386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:44.046711 26386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:44.050253 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "cdd45b89e8b54e90ad1a8194cb291aca"
format_stamp: "Formatted at 2026-06-19 16:56:44 on dist-test-slave-3ffd"
server_key: "aefca7720937fe44f3439f42c3170e39"
server_key_iv: "60a8ee93686a502f2894a6afe79361d1"
server_key_version: "encryptionkey@0"
I20260619 16:56:44.051100 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "cdd45b89e8b54e90ad1a8194cb291aca"
format_stamp: "Formatted at 2026-06-19 16:56:44 on dist-test-slave-3ffd"
server_key: "aefca7720937fe44f3439f42c3170e39"
server_key_iv: "60a8ee93686a502f2894a6afe79361d1"
server_key_version: "encryptionkey@0"
I20260619 16:56:44.056540 26386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:56:44.060263 26400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:44.061721 26386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:44.061928 26386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "cdd45b89e8b54e90ad1a8194cb291aca"
format_stamp: "Formatted at 2026-06-19 16:56:44 on dist-test-slave-3ffd"
server_key: "aefca7720937fe44f3439f42c3170e39"
server_key_iv: "60a8ee93686a502f2894a6afe79361d1"
server_key_version: "encryptionkey@0"
I20260619 16:56:44.062170 26386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:44.068490 26386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:44.069382 26386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:44.069684 26386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:44.088419 26386 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:44689
I20260619 16:56:44.088441 26451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:44689 every 8 connection(s)
I20260619 16:56:44.089874 26386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:56:44.093722 26452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:44.100442 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 26386
I20260619 16:56:44.100684 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:56:44.101312 16576 external_mini_cluster.cc:1428] Setting key 84d68d58231dd46ed969b568e93d2413
I20260619 16:56:44.105544 26452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca: Bootstrap starting.
I20260619 16:56:44.109436 26452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:44.110549 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:44.103674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.16.48.62:44689)
0619 16:56:44.104792 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:44.104812 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:44.104857 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:56:44.106607 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:44.106633 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:44.107102 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:44.107490 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:44.108717 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:44.108724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:44.108738 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:44.108752 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:44.108821 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:44.108838 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:44.108995 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:44.109169 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 16:56:44.109499 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":412,"threads_started":1}
I20260619 16:56:44.110600 26452 log.cc:824] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:44.113852 26452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca: No bootstrap required, opened a new log
I20260619 16:56:44.122994 26452 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd45b89e8b54e90ad1a8194cb291aca" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44689 } }
I20260619 16:56:44.123420 26452 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:44.123549 26452 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd45b89e8b54e90ad1a8194cb291aca, State: Initialized, Role: FOLLOWER
I20260619 16:56:44.124191 26452 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd45b89e8b54e90ad1a8194cb291aca" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44689 } }
I20260619 16:56:44.124454 26452 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:56:44.124608 26452 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:56:44.124801 26452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:44.127079 26452 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd45b89e8b54e90ad1a8194cb291aca" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44689 } }
I20260619 16:56:44.127637 26452 leader_election.cc:302] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd45b89e8b54e90ad1a8194cb291aca; no voters:
I20260619 16:56:44.128322 26452 leader_election.cc:288] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:56:44.128458 26457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:44.129200 26457 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [term 1 LEADER]: Becoming Leader. State: Replica: cdd45b89e8b54e90ad1a8194cb291aca, State: Running, Role: LEADER
I20260619 16:56:44.129799 26457 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd45b89e8b54e90ad1a8194cb291aca" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44689 } }
I20260619 16:56:44.130614 26452 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:56:44.132704 26458 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd45b89e8b54e90ad1a8194cb291aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd45b89e8b54e90ad1a8194cb291aca" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44689 } } }
I20260619 16:56:44.133046 26458 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:44.133904 26462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:56:44.133764 26459 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd45b89e8b54e90ad1a8194cb291aca. Latest consensus state: current_term: 1 leader_uuid: "cdd45b89e8b54e90ad1a8194cb291aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd45b89e8b54e90ad1a8194cb291aca" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44689 } } }
I20260619 16:56:44.134093 26459 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:44.139987 26462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:56:44.152207 26462 catalog_manager.cc:1347] Generated new cluster ID: 9fa517ca61e742388289686f98e2db3e
I20260619 16:56:44.152351 26462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:56:44.185899 26462 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:56:44.186440 26462 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:56:44.201122 26462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd45b89e8b54e90ad1a8194cb291aca: Generated new TSK 0
I20260619 16:56:44.202267 26462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:56:44.209498 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44689
--builtin_ntp_servers=127.16.48.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:56:44.562399 26476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:44.562714 26476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:44.562809 26476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:44.570597 26476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:44.570725 26476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:44.570837 26476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:56:44.580469 26476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:44.581759 26476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:44.583532 26476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:44.597754 26481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:44.597836 26482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:44.597791 26484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:44.601950 26476 server_base.cc:1034] running on GCE node
I20260619 16:56:44.602716 26476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:44.603804 26476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:44.605069 26476 hybrid_clock.cc:648] HybridClock initialized: now 1781888204604991 us; error 80 us; skew 500 ppm
I20260619 16:56:44.605441 26476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:56:44.608031 26476 webserver.cc:460] Webserver started at http://127.16.48.1:34737/ using document root <none> and password file <none>
I20260619 16:56:44.608877 26476 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:44.609033 26476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:44.609381 26476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:44.612790 26476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "4b57d230a4414fd093202e1c0508de5e"
format_stamp: "Formatted at 2026-06-19 16:56:44 on dist-test-slave-3ffd"
server_key: "e0d4eb7e882dc384ce7d0840c17c17fd"
server_key_iv: "1b3f9710ef4f0b1baf2417221986a6ba"
server_key_version: "encryptionkey@0"
I20260619 16:56:44.613600 26476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "4b57d230a4414fd093202e1c0508de5e"
format_stamp: "Formatted at 2026-06-19 16:56:44 on dist-test-slave-3ffd"
server_key: "e0d4eb7e882dc384ce7d0840c17c17fd"
server_key_iv: "1b3f9710ef4f0b1baf2417221986a6ba"
server_key_version: "encryptionkey@0"
I20260619 16:56:44.619261 26476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:56:44.623170 26490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:44.624778 26476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:56:44.624976 26476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "4b57d230a4414fd093202e1c0508de5e"
format_stamp: "Formatted at 2026-06-19 16:56:44 on dist-test-slave-3ffd"
server_key: "e0d4eb7e882dc384ce7d0840c17c17fd"
server_key_iv: "1b3f9710ef4f0b1baf2417221986a6ba"
server_key_version: "encryptionkey@0"
I20260619 16:56:44.625229 26476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:44.631194 26476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:44.632045 26476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:44.632377 26476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:44.633373 26476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:44.634865 26476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:44.634956 26476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:44.635049 26476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:44.635104 26476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:44.669548 26476 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:34243
I20260619 16:56:44.669642 26602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:34243 every 8 connection(s)
I20260619 16:56:44.670902 26476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:56:44.673676 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 26476
I20260619 16:56:44.673906 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:56:44.674642 16576 external_mini_cluster.cc:1428] Setting key cafec154a207e9aee457226aeb563dd7
I20260619 16:56:44.677428 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44689
--builtin_ntp_servers=127.16.48.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:56:44.680076 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:44.673631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:46285 (local address 127.16.48.62:44689)
0619 16:56:44.674201 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:44.674209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:44.675701 (+ 1492us) server_negotiation.cc:407] Connection header received
0619 16:56:44.677538 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:44.677543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:44.677614 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:44.677745 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:44.679265 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:44.679270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:44.679274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:44.679291 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:44.679334 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:44.679346 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:44.679485 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:44.679699 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 16:56:44.679772 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":285,"threads_started":1}
I20260619 16:56:44.681149 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:44.674366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44689 (local address 127.16.48.1:46285)
0619 16:56:44.675518 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:44.675575 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:44.677226 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:44.678033 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:44.678042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:44.678155 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:44.678918 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:56:44.678920 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:44.678960 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:56:44.679070 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:44.679453 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:44.679465 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:44.679473 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:56:44.679780 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 16:56:44.680095 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":353,"threads_started":1}
I20260619 16:56:44.682917 26603 heartbeater.cc:347] Connected to a master server at 127.16.48.62:44689
I20260619 16:56:44.683241 26603 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:44.683869 26603 heartbeater.cc:511] Master 127.16.48.62:44689 requested a full tablet report, sending...
I20260619 16:56:44.686326 26417 ts_manager.cc:194] Registered new tserver with Master: 4b57d230a4414fd093202e1c0508de5e (127.16.48.1:34243)
W20260619 16:56:45.027014 26607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:45.027325 26607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:45.027415 26607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:45.035189 26607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:45.035313 26607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:45.035429 26607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:56:45.045094 26607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:45.046429 26607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:45.048303 26607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:45.062311 26612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:45.062292 26615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:45.062382 26613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:45.067224 26607 server_base.cc:1034] running on GCE node
I20260619 16:56:45.067929 26607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:45.069048 26607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:45.070312 26607 hybrid_clock.cc:648] HybridClock initialized: now 1781888205070261 us; error 52 us; skew 500 ppm
I20260619 16:56:45.070688 26607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:56:45.073356 26607 webserver.cc:460] Webserver started at http://127.16.48.2:46051/ using document root <none> and password file <none>
I20260619 16:56:45.074132 26607 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:45.074275 26607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:45.074623 26607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:45.077759 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "5b8b50a4528e40e494233b5f4633d72a"
format_stamp: "Formatted at 2026-06-19 16:56:45 on dist-test-slave-3ffd"
server_key: "15c3aacf17d3fd5dcd63a6262182dc55"
server_key_iv: "397e089791396ee45b22fa4c34a74558"
server_key_version: "encryptionkey@0"
I20260619 16:56:45.078657 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "5b8b50a4528e40e494233b5f4633d72a"
format_stamp: "Formatted at 2026-06-19 16:56:45 on dist-test-slave-3ffd"
server_key: "15c3aacf17d3fd5dcd63a6262182dc55"
server_key_iv: "397e089791396ee45b22fa4c34a74558"
server_key_version: "encryptionkey@0"
I20260619 16:56:45.084462 26607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 16:56:45.088419 26621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:45.090082 26607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:45.090281 26607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "5b8b50a4528e40e494233b5f4633d72a"
format_stamp: "Formatted at 2026-06-19 16:56:45 on dist-test-slave-3ffd"
server_key: "15c3aacf17d3fd5dcd63a6262182dc55"
server_key_iv: "397e089791396ee45b22fa4c34a74558"
server_key_version: "encryptionkey@0"
I20260619 16:56:45.090533 26607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:45.097078 26607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:45.097987 26607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:45.098297 26607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:45.099431 26607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:45.101225 26607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:45.101320 26607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:45.101480 26607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:45.101574 26607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:45.137096 26607 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:38169
I20260619 16:56:45.137219 26733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:38169 every 8 connection(s)
I20260619 16:56:45.138614 26607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:56:45.143894 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 26607
I20260619 16:56:45.144109 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:56:45.144603 16576 external_mini_cluster.cc:1428] Setting key 3fe980e53df9d777e7498c0c0ba8f67f
I20260619 16:56:45.147665 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44689
--builtin_ntp_servers=127.16.48.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:56:45.148025 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:45.141414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:38879 (local address 127.16.48.62:44689)
0619 16:56:45.141635 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:45.141642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:45.143302 (+ 1660us) server_negotiation.cc:407] Connection header received
0619 16:56:45.145400 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:45.145408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:45.145477 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:45.145601 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:45.147242 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:45.147246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:45.147249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:45.147265 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:45.147297 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:45.147308 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:45.147442 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:45.147675 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 16:56:45.147745 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:56:45.149147 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:45.142090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44689 (local address 127.16.48.2:38879)
0619 16:56:45.143122 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:45.143167 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:45.144992 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:45.145918 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:45.145932 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:45.146075 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:45.146876 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:56:45.146880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:45.146936 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:56:45.147034 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:45.147405 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:45.147417 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:45.147427 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:56:45.147766 (+ 339us) client_negotiation.cc:240] Negotiation successful
0619 16:56:45.148073 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":340,"threads_started":1}
I20260619 16:56:45.150861 26734 heartbeater.cc:347] Connected to a master server at 127.16.48.62:44689
I20260619 16:56:45.151296 26734 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:45.152007 26734 heartbeater.cc:511] Master 127.16.48.62:44689 requested a full tablet report, sending...
I20260619 16:56:45.153786 26417 ts_manager.cc:194] Registered new tserver with Master: 5b8b50a4528e40e494233b5f4633d72a (127.16.48.2:38169)
W20260619 16:56:45.495565 26737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:45.495957 26737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:45.496173 26737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:45.503859 26737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:45.504025 26737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:45.504243 26737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:56:45.513833 26737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:45.515245 26737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:45.517201 26737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:45.531563 26742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:45.531647 26745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:45.531607 26743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:45.532613 26737 server_base.cc:1034] running on GCE node
I20260619 16:56:45.533362 26737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:45.534431 26737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:45.535687 26737 hybrid_clock.cc:648] HybridClock initialized: now 1781888205535640 us; error 46 us; skew 500 ppm
I20260619 16:56:45.536049 26737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:56:45.538614 26737 webserver.cc:460] Webserver started at http://127.16.48.3:33183/ using document root <none> and password file <none>
I20260619 16:56:45.539399 26737 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:45.539541 26737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:45.539886 26737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:45.543207 26737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "b27c63b152e0433d8ca8a4b9f872138a"
format_stamp: "Formatted at 2026-06-19 16:56:45 on dist-test-slave-3ffd"
server_key: "17bc9661ac2c754e21f21f63f5dad08f"
server_key_iv: "735fae9c1cf1aa6c13eb0eeb88e68cc7"
server_key_version: "encryptionkey@0"
I20260619 16:56:45.544109 26737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "b27c63b152e0433d8ca8a4b9f872138a"
format_stamp: "Formatted at 2026-06-19 16:56:45 on dist-test-slave-3ffd"
server_key: "17bc9661ac2c754e21f21f63f5dad08f"
server_key_iv: "735fae9c1cf1aa6c13eb0eeb88e68cc7"
server_key_version: "encryptionkey@0"
I20260619 16:56:45.549890 26737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:56:45.553784 26751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:45.555336 26737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:56:45.555644 26737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "b27c63b152e0433d8ca8a4b9f872138a"
format_stamp: "Formatted at 2026-06-19 16:56:45 on dist-test-slave-3ffd"
server_key: "17bc9661ac2c754e21f21f63f5dad08f"
server_key_iv: "735fae9c1cf1aa6c13eb0eeb88e68cc7"
server_key_version: "encryptionkey@0"
I20260619 16:56:45.555896 26737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:45.562079 26737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:45.563066 26737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:45.563364 26737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:45.564435 26737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:45.566073 26737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:45.566203 26737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:45.566336 26737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:45.566411 26737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:45.600750 26737 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:44081
I20260619 16:56:45.600800 26863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:44081 every 8 connection(s)
I20260619 16:56:45.602288 26737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:56:45.603506 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 26737
I20260619 16:56:45.603701 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:56:45.604285 16576 external_mini_cluster.cc:1428] Setting key 3d96bc4b86065f640bd83549dff0faa5
I20260619 16:56:45.611496 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:45.605415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:37557 (local address 127.16.48.62:44689)
0619 16:56:45.605612 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:45.605617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:45.607618 (+ 2001us) server_negotiation.cc:407] Connection header received
0619 16:56:45.609405 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:45.609409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:45.609463 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:45.609552 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:45.610909 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:45.610913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:45.610916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:45.610930 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:45.610954 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:45.610965 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:45.611063 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:45.611236 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 16:56:45.611284 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:56:45.612632 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:45.606137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44689 (local address 127.16.48.3:37557)
0619 16:56:45.607444 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:45.607496 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:45.609104 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:45.609768 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:45.609777 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:45.609875 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:45.610608 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:56:45.610610 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:45.610654 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:56:45.610747 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:45.611055 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:45.611062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:45.611067 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:56:45.611285 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 16:56:45.611505 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":384,"threads_started":1}
I20260619 16:56:45.614187 26864 heartbeater.cc:347] Connected to a master server at 127.16.48.62:44689
I20260619 16:56:45.614537 26864 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:45.615165 26864 heartbeater.cc:511] Master 127.16.48.62:44689 requested a full tablet report, sending...
I20260619 16:56:45.616719 26417 ts_manager.cc:194] Registered new tserver with Master: b27c63b152e0433d8ca8a4b9f872138a (127.16.48.3:44081)
I20260619 16:56:45.619940 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:56:45.626154 26605 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:56:45.624589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.16.48.62:44689)
0619 16:56:45.624760 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:45.624765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:45.625322 (+ 557us) server_negotiation.cc:407] Connection header received
0619 16:56:45.625470 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:45.625474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:45.625532 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:45.625622 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:45.625948 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40416: BlockingRecv error: recv got EOF from 127.0.0.1:40416 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:56:45.626766 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 26476
I20260619 16:56:45.630793 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:45.628179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.16.48.1:34243)
0619 16:56:45.628771 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:45.628785 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:45.628811 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:56:45.628870 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:45.628876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:45.629153 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:45.629281 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:45.630193 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:45.630199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:45.630212 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:45.630230 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:45.630271 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:45.630285 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:45.630389 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:45.630468 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 16:56:45.630542 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":300,"threads_started":1}
I20260619 16:56:45.632400 26598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:45.714582 26603 heartbeater.cc:503] Master 127.16.48.62:44689 was elected leader, sending a full tablet report...
I20260619 16:56:45.716461 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 26476
I20260619 16:56:45.733088 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 26607
I20260619 16:56:45.737000 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:45.734072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.16.48.2:38169)
0619 16:56:45.734651 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:45.734658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:45.734686 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:56:45.734749 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:45.734755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:45.734995 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:45.735132 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:45.736174 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:45.736185 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:45.736199 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:45.736241 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:45.736287 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:45.736304 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:45.736426 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:45.736604 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:56:45.736701 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":287,"threads_started":1}
I20260619 16:56:45.738453 26729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:45.821306 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 26607
I20260619 16:56:45.839097 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 26737
I20260619 16:56:45.843468 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:45.840056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.16.48.3:44081)
0619 16:56:45.840803 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:45.840810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:45.840841 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:56:45.840902 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:45.840908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:45.841149 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:45.841354 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:45.842587 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:45.842595 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:45.842607 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:56:45.842624 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:45.842662 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:56:45.842677 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:45.842868 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:45.842944 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:56:45.843046 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":458,"threads_started":1}
I20260619 16:56:45.844941 26859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:45.927596 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 26737
I20260619 16:56:45.943743 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 26386
I20260619 16:56:45.945092 26447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:46.047281 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 26386
2026-06-19T16:56:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2465 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:56:46 dist-test-slave-3ffd krb5kdc[26882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:56:46 dist-test-slave-3ffd krb5kdc[26882](info): set up 2 sockets
Jun 19 16:56:46 dist-test-slave-3ffd krb5kdc[26882](info): commencing operation
krb5kdc: starting...
W20260619 16:56:47.165498 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:56:47 dist-test-slave-3ffd krb5kdc[26882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:56:47.419420 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:45255
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:45255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:47.764446 26895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:47.764763 26895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:47.764953 26895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:47.772459 26895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:56:47.772580 26895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:47.772694 26895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:47.772774 26895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:56:47.772845 26895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:56:47.783213 26895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:45255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:45255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:47.784683 26895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:47.786093 26895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:47.798316 26903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:47.798448 26901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:47.798413 26900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:47.799194 26895 server_base.cc:1034] running on GCE node
I20260619 16:56:47.800094 26895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:47.801752 26895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:47.802992 26895 hybrid_clock.cc:648] HybridClock initialized: now 1781888207802924 us; error 66 us; skew 500 ppm
I20260619 16:56:47.803366 26895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:47 dist-test-slave-3ffd krb5kdc[26882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888207, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:47.812393 26895 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:56:47.814198 26895 webserver.cc:460] Webserver started at http://127.16.48.62:38567/ using document root <none> and password file <none>
I20260619 16:56:47.814983 26895 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:47.815147 26895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:47.815488 26895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:47.818924 26895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0"
format_stamp: "Formatted at 2026-06-19 16:56:47 on dist-test-slave-3ffd"
server_key: "6fb44f8c9fecb85b9b963afd0fd7c963"
server_key_iv: "7c71f255ef51dbe44fde2715414dc864"
server_key_version: "encryptionkey@0"
I20260619 16:56:47.819810 26895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0"
format_stamp: "Formatted at 2026-06-19 16:56:47 on dist-test-slave-3ffd"
server_key: "6fb44f8c9fecb85b9b963afd0fd7c963"
server_key_iv: "7c71f255ef51dbe44fde2715414dc864"
server_key_version: "encryptionkey@0"
I20260619 16:56:47.825759 26895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 16:56:47.829504 26910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:47.830987 26895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:56:47.831195 26895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0"
format_stamp: "Formatted at 2026-06-19 16:56:47 on dist-test-slave-3ffd"
server_key: "6fb44f8c9fecb85b9b963afd0fd7c963"
server_key_iv: "7c71f255ef51dbe44fde2715414dc864"
server_key_version: "encryptionkey@0"
I20260619 16:56:47.831411 26895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:47.847364 26895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:47.851862 26895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:47.852238 26895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:47.871764 26895 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:45255
I20260619 16:56:47.871795 26961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:45255 every 8 connection(s)
I20260619 16:56:47.873288 26895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:56:47.874578 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 26895
I20260619 16:56:47.874815 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:56:47.875437 16576 external_mini_cluster.cc:1428] Setting key 459e65a6b5c69271b1bc10d725fde349
I20260619 16:56:47.877854 26962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:56:47 dist-test-slave-3ffd krb5kdc[26882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:56:47.891752 26962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0: Bootstrap starting.
I20260619 16:56:47.896183 26962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:47.897617 26962 log.cc:824] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:47.901017 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:47.877468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.16.48.62:45255)
0619 16:56:47.878276 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:47.878290 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:47.878332 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:56:47.879908 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:47.879929 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:47.880421 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:47.880789 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:47.882757 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:47.884215 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:47.886000 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:47.886694 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:47.886833 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:47.891552 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:47.891564 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:47.891585 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:47.891631 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:47.895595 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:47.896660 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:47.896675 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:47.896688 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:47.896909 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:47.897352 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:47.897359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:47.897364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:47.898131 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:47.898500 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:47.899080 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 16:56:47.899579 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":377,"threads_started":1}
I20260619 16:56:47.901468 26962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0: No bootstrap required, opened a new log
I20260619 16:56:47.909670 26962 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45255 } }
I20260619 16:56:47.910166 26962 raft_consensus.cc:374] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:47.910293 26962 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71c52a7d7ff412fbcd109fbdb05d4c0, State: Initialized, Role: FOLLOWER
I20260619 16:56:47.910996 26962 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45255 } }
I20260619 16:56:47.911267 26962 raft_consensus.cc:388] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:56:47.911413 26962 raft_consensus.cc:482] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:56:47.911595 26962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:47.913944 26962 raft_consensus.cc:504] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45255 } }
I20260619 16:56:47.914489 26962 leader_election.cc:302] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f71c52a7d7ff412fbcd109fbdb05d4c0; no voters:
I20260619 16:56:47.915269 26962 leader_election.cc:288] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:56:47.915408 26967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:47.916260 26967 raft_consensus.cc:686] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [term 1 LEADER]: Becoming Leader. State: Replica: f71c52a7d7ff412fbcd109fbdb05d4c0, State: Running, Role: LEADER
I20260619 16:56:47.916924 26967 consensus_queue.cc:237] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45255 } }
I20260619 16:56:47.917665 26962 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:56:47.919832 26968 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45255 } } }
I20260619 16:56:47.921782 26968 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:47.921379 26969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f71c52a7d7ff412fbcd109fbdb05d4c0. Latest consensus state: current_term: 1 leader_uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71c52a7d7ff412fbcd109fbdb05d4c0" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45255 } } }
I20260619 16:56:47.921924 26969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:56:47.924876 26978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:56:47.931334 26978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:56:47.940714 26978 catalog_manager.cc:1347] Generated new cluster ID: a9302b8b4dd24153b272f37e7e3adfd7
I20260619 16:56:47.940845 26978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:56:47.966441 26978 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:56:47.967967 26978 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:56:47.983789 26978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0: Generated new TSK 0
I20260619 16:56:47.984841 26978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:56:48.062278 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:45255
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:48.451709 26988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:48.452013 26988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:48.452212 26988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:48.459591 26988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:48.459723 26988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:48.459921 26988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:56:48.470065 26988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:45255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:48.471541 26988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:48.473069 26988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:48.487360 26996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:48.487324 26994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:48.487404 26993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:48.489503 26988 server_base.cc:1034] running on GCE node
I20260619 16:56:48.490170 26988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:48.491153 26988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:48.492352 26988 hybrid_clock.cc:648] HybridClock initialized: now 1781888208492304 us; error 38 us; skew 500 ppm
I20260619 16:56:48.492695 26988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:48 dist-test-slave-3ffd krb5kdc[26882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888208, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:48.495970 26988 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:56:48.497670 26988 webserver.cc:460] Webserver started at http://127.16.48.1:35179/ using document root <none> and password file <none>
I20260619 16:56:48.498445 26988 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:48.498572 26988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:48.498896 26988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:48.502271 26988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "02a254f865a549fb8b2d388d3e5a1190"
format_stamp: "Formatted at 2026-06-19 16:56:48 on dist-test-slave-3ffd"
server_key: "5d953e9317b6a6b389f0f38ce8d958a4"
server_key_iv: "0c85963d9b8afe39ab64fc6d0f21b891"
server_key_version: "encryptionkey@0"
I20260619 16:56:48.503091 26988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "02a254f865a549fb8b2d388d3e5a1190"
format_stamp: "Formatted at 2026-06-19 16:56:48 on dist-test-slave-3ffd"
server_key: "5d953e9317b6a6b389f0f38ce8d958a4"
server_key_iv: "0c85963d9b8afe39ab64fc6d0f21b891"
server_key_version: "encryptionkey@0"
I20260619 16:56:48.508807 26988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:56:48.513082 27003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:48.514801 26988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:56:48.515026 26988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "02a254f865a549fb8b2d388d3e5a1190"
format_stamp: "Formatted at 2026-06-19 16:56:48 on dist-test-slave-3ffd"
server_key: "5d953e9317b6a6b389f0f38ce8d958a4"
server_key_iv: "0c85963d9b8afe39ab64fc6d0f21b891"
server_key_version: "encryptionkey@0"
I20260619 16:56:48.515266 26988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:48.537739 26988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:48.542217 26988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:48.542557 26988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:48.543641 26988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:48.545284 26988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:48.545408 26988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:48.545567 26988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:48.545640 26988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:48.580685 26988 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:44989
I20260619 16:56:48.580745 27115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:44989 every 8 connection(s)
I20260619 16:56:48.582080 26988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:56:48.584005 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 26988
I20260619 16:56:48.584259 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:56:48.585098 16576 external_mini_cluster.cc:1428] Setting key 77bf14b93d9c8c99a3dad9a6c2f3728e
Jun 19 16:56:48 dist-test-slave-3ffd krb5kdc[26882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888208, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:56:48.605247 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:48.585262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:58215 (local address 127.16.48.62:45255)
0619 16:56:48.585781 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:48.585787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:48.587054 (+ 1267us) server_negotiation.cc:407] Connection header received
0619 16:56:48.589098 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:48.589107 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:48.589201 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:48.589330 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:48.591381 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:48.592568 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:48.594832 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:48.595238 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:48.595383 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:48.599083 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:48.599088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:48.599091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:48.599127 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:48.601946 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:48.602746 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:48.602758 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:48.602761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:48.602878 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:48.603330 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:48.603336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:48.603339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:48.603634 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:48.603793 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:48.604612 (+ 819us) server_negotiation.cc:299] Negotiation successful
0619 16:56:48.604852 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":262,"threads_started":1}
I20260619 16:56:48.606415 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:48.585964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:45255 (local address 127.16.48.1:58215)
0619 16:56:48.586898 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:48.586946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:48.588781 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:48.589610 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:48.589625 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:48.590374 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:48.591161 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:48.591186 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:48.592745 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:48.592752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:48.594646 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:48.594659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:48.595491 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:48.595498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:48.595750 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:48.596542 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:48.596571 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:48.598802 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:48.602104 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:48.602114 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:48.602143 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:48.602602 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:48.602998 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:48.603003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:48.603007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:48.603207 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:48.603772 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:48.603780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:48.604220 (+ 440us) client_negotiation.cc:769] Sending connection context
0619 16:56:48.604516 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 16:56:48.605108 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":406,"threads_started":1}
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 16:56:48.608206 27116 heartbeater.cc:347] Connected to a master server at 127.16.48.62:45255
I20260619 16:56:48.608551 27116 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:48.609349 27116 heartbeater.cc:511] Master 127.16.48.62:45255 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:56:48.611833 26927 ts_manager.cc:194] Registered new tserver with Master: 02a254f865a549fb8b2d388d3e5a1190 (127.16.48.1:44989)
I20260619 16:56:48.614096 26927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:58215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:56:48.641269 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:45255
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:48.991976 27122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:48.992376 27122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:48.992488 27122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:49.000188 27122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:49.000339 27122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:49.000473 27122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:56:49.011232 27122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:45255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:49.012804 27122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:49.014165 27122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:49.029254 27130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:49.029281 27127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:49.029335 27128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:49.030192 27122 server_base.cc:1034] running on GCE node
I20260619 16:56:49.030877 27122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:49.031940 27122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:49.033217 27122 hybrid_clock.cc:648] HybridClock initialized: now 1781888209033140 us; error 67 us; skew 500 ppm
I20260619 16:56:49.033668 27122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:49 dist-test-slave-3ffd krb5kdc[26882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888209, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:49.037477 27122 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:56:49.039182 27122 webserver.cc:460] Webserver started at http://127.16.48.2:36437/ using document root <none> and password file <none>
I20260619 16:56:49.039963 27122 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:49.040172 27122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:49.040520 27122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:49.043712 27122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "3d22fe11a0bb412c956ff8d5282dd15c"
format_stamp: "Formatted at 2026-06-19 16:56:49 on dist-test-slave-3ffd"
server_key: "a78c7b3c76079a231334414391a70455"
server_key_iv: "646d6c3532176bddf9f7613545219a43"
server_key_version: "encryptionkey@0"
I20260619 16:56:49.044574 27122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "3d22fe11a0bb412c956ff8d5282dd15c"
format_stamp: "Formatted at 2026-06-19 16:56:49 on dist-test-slave-3ffd"
server_key: "a78c7b3c76079a231334414391a70455"
server_key_iv: "646d6c3532176bddf9f7613545219a43"
server_key_version: "encryptionkey@0"
I20260619 16:56:49.050351 27122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 16:56:49.054569 27137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:49.056244 27122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:56:49.056485 27122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "3d22fe11a0bb412c956ff8d5282dd15c"
format_stamp: "Formatted at 2026-06-19 16:56:49 on dist-test-slave-3ffd"
server_key: "a78c7b3c76079a231334414391a70455"
server_key_iv: "646d6c3532176bddf9f7613545219a43"
server_key_version: "encryptionkey@0"
I20260619 16:56:49.056715 27122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:49.081229 27122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:49.086158 27122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:49.086503 27122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:49.087556 27122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:49.089092 27122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:49.089197 27122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:49.089356 27122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:49.089430 27122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:49.124038 27122 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:44987
I20260619 16:56:49.124174 27249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:44987 every 8 connection(s)
I20260619 16:56:49.125406 27122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:56:49.125917 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 27122
I20260619 16:56:49.126123 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:56:49.126721 16576 external_mini_cluster.cc:1428] Setting key 8da651165c2db009391e6b69bb8d2e7f
Jun 19 16:56:49 dist-test-slave-3ffd krb5kdc[26882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888209, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:56:49.149428 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.128216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:55217 (local address 127.16.48.62:45255)
0619 16:56:49.128699 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:49.128705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:49.130017 (+ 1312us) server_negotiation.cc:407] Connection header received
0619 16:56:49.132075 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:49.132081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:49.132232 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:49.132377 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:49.134168 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.135066 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.137046 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.137526 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.137690 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.142516 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:49.142523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:49.142527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:49.142568 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:49.145806 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:49.146732 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:49.146738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:49.146742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:49.146892 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:49.147433 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:49.147439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:49.147441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:49.147759 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:49.147934 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:49.148782 (+ 848us) server_negotiation.cc:299] Negotiation successful
0619 16:56:49.149024 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":255,"threads_started":1}
I20260619 16:56:49.150529 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.128948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:45255 (local address 127.16.48.2:55217)
0619 16:56:49.129844 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:49.129891 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:49.131736 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:49.132666 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:49.132676 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:49.133259 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:49.133948 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.133969 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.135260 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.135264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.136867 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.136884 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.137799 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.137807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.138173 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.139163 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:49.139201 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:49.142187 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:49.146023 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:49.146035 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:49.146062 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:49.146569 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:49.147028 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:49.147034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:49.147038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:49.147293 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:49.147917 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:49.147926 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:49.148395 (+ 469us) client_negotiation.cc:769] Sending connection context
0619 16:56:49.148677 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 16:56:49.149288 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":386,"threads_started":1}
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
I20260619 16:56:49.152298 27250 heartbeater.cc:347] Connected to a master server at 127.16.48.62:45255
I20260619 16:56:49.152657 27250 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:49.153431 27250 heartbeater.cc:511] Master 127.16.48.62:45255 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
I20260619 16:56:49.155202 26927 ts_manager.cc:194] Registered new tserver with Master: 3d22fe11a0bb412c956ff8d5282dd15c (127.16.48.2:44987)
I20260619 16:56:49.156255 26927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:55217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:56:49.187876 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:45255
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:56:49.532624 27256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:56:49.532919 27256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:56:49.533102 27256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:56:49.540759 27256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:56:49.540884 27256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:56:49.541075 27256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:56:49.551282 27256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:45255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:56:49.552738 27256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:56:49.554287 27256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:56:49.568744 27261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:49.568768 27262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:56:49.568792 27264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:56:49.569312 27256 server_base.cc:1034] running on GCE node
I20260619 16:56:49.571448 27256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:56:49.572539 27256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:56:49.573776 27256 hybrid_clock.cc:648] HybridClock initialized: now 1781888209573733 us; error 45 us; skew 500 ppm
I20260619 16:56:49.574162 27256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:56:49 dist-test-slave-3ffd krb5kdc[26882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888209, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:56:49.577874 27256 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:56:49.579586 27256 webserver.cc:460] Webserver started at http://127.16.48.3:37907/ using document root <none> and password file <none>
I20260619 16:56:49.580484 27256 fs_manager.cc:359] Metadata directory not provided
I20260619 16:56:49.580637 27256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:56:49.581002 27256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:56:49.584084 27256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "f941aa85a3b74525ba6b7a3bbfd18850"
format_stamp: "Formatted at 2026-06-19 16:56:49 on dist-test-slave-3ffd"
server_key: "269ddb99f0c07517b3f3ec25c950db19"
server_key_iv: "664ad021481a610ed8b7e95e0aa20122"
server_key_version: "encryptionkey@0"
I20260619 16:56:49.585119 27256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "f941aa85a3b74525ba6b7a3bbfd18850"
format_stamp: "Formatted at 2026-06-19 16:56:49 on dist-test-slave-3ffd"
server_key: "269ddb99f0c07517b3f3ec25c950db19"
server_key_iv: "664ad021481a610ed8b7e95e0aa20122"
server_key_version: "encryptionkey@0"
I20260619 16:56:49.590898 27256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:56:49.594842 27271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:49.596453 27256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:56:49.596652 27256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "f941aa85a3b74525ba6b7a3bbfd18850"
format_stamp: "Formatted at 2026-06-19 16:56:49 on dist-test-slave-3ffd"
server_key: "269ddb99f0c07517b3f3ec25c950db19"
server_key_iv: "664ad021481a610ed8b7e95e0aa20122"
server_key_version: "encryptionkey@0"
I20260619 16:56:49.596875 27256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:56:49.613906 27256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:56:49.617693 27116 heartbeater.cc:503] Master 127.16.48.62:45255 was elected leader, sending a full tablet report...
I20260619 16:56:49.619022 27256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:56:49.619397 27256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:56:49.620586 27256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:56:49.622109 27256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:56:49.622193 27256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:49.622283 27256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:56:49.622323 27256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:56:49.656654 27256 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:41207
I20260619 16:56:49.656710 27383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:41207 every 8 connection(s)
I20260619 16:56:49.658036 27256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:56:49.662794 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 27256
I20260619 16:56:49.663025 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:56:49.663466 16576 external_mini_cluster.cc:1428] Setting key 0cb7f1b3daea5f3d99d9c60fe37af133
Jun 19 16:56:49 dist-test-slave-3ffd krb5kdc[26882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888209, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:56:49.680787 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.660871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:47837 (local address 127.16.48.62:45255)
0619 16:56:49.661309 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:49.661313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:49.662514 (+ 1201us) server_negotiation.cc:407] Connection header received
0619 16:56:49.664790 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:49.664797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:49.664906 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:49.665041 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:49.667024 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.667804 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.669494 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.669913 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.670027 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.673839 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:49.673845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:49.673848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:49.673885 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:49.676791 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:49.677895 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:49.677900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:49.677903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:49.678085 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:49.678689 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:49.678695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:49.678698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:49.679176 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:49.679329 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:49.680282 (+ 953us) server_negotiation.cc:299] Negotiation successful
0619 16:56:49.680500 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260619 16:56:49.681767 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.661506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:45255 (local address 127.16.48.3:47837)
0619 16:56:49.662327 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:49.662375 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:49.664414 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:49.665278 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:49.665287 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:49.666078 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:56:49.666869 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.666889 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.667941 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.667944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.669312 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.669325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.670158 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.670164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.670386 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.671141 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:56:49.671167 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:56:49.673588 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:56:49.677029 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:49.677037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:49.677055 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:49.677752 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:49.678237 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:56:49.678241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:56:49.678243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:56:49.678524 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:56:49.679351 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:56:49.679360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:56:49.679812 (+ 452us) client_negotiation.cc:769] Sending connection context
0619 16:56:49.680068 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 16:56:49.680593 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":373,"threads_started":1}
I20260619 16:56:49.683409 27384 heartbeater.cc:347] Connected to a master server at 127.16.48.62:45255
I20260619 16:56:49.683754 27384 heartbeater.cc:464] Registering TS with master...
I20260619 16:56:49.684624 27384 heartbeater.cc:511] Master 127.16.48.62:45255 requested a full tablet report, sending...
I20260619 16:56:49.686260 26927 ts_manager.cc:194] Registered new tserver with Master: f941aa85a3b74525ba6b7a3bbfd18850 (127.16.48.3:41207)
I20260619 16:56:49.687171 26927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:47837
I20260619 16:56:49.692063 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:56:49.710726 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.697006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.16.48.62:45255)
0619 16:56:49.697181 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:49.697186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:49.697724 (+ 538us) server_negotiation.cc:407] Connection header received
0619 16:56:49.697876 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:49.697880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:49.697944 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:49.698039 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:49.699609 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.700528 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.702047 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.702457 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.702559 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.705239 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:49.705244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:49.705248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:49.705285 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:49.707875 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:49.708737 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:49.708741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:49.708743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:49.708872 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:49.709291 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:49.709295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:49.709297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:49.709644 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:49.709825 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:49.710281 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 16:56:49.710465 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:56:49.715171 26927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:56:49.722422 26927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:56:49.753567 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.735455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.16.48.2:44987)
0619 16:56:49.743375 (+ 7920us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:49.743389 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:49.743430 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:56:49.743575 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:49.743587 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:49.744066 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:49.744355 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:49.745772 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.747241 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.750334 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.751932 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.752062 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.752339 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:49.753012 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 16:56:49.753220 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7690,"thread_start_us":7556,"threads_started":1}
I20260619 16:56:49.754662 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.735439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:44987 (local address 127.0.0.1:59890)
0619 16:56:49.742527 (+ 7088us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:49.742585 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:49.742797 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:49.744750 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:49.744763 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:49.744812 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:49.745477 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.745503 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.747425 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.747430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.750127 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.750141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.752517 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.752524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.752724 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.752787 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 16:56:49.753993 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0619 16:56:49.754256 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6819,"thread_start_us":6582,"threads_started":1}
I20260619 16:56:49.756446 27399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.735748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:44989 (local address 127.0.0.1:56196)
0619 16:56:49.746153 (+ 10405us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:49.746174 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:49.746312 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:49.747208 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:49.747224 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:49.747257 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:49.747782 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.747794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.749834 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.749839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.752030 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.752048 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.754327 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.754333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.754536 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.754570 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:56:49.754678 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 16:56:49.754838 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10270,"mutex_wait_us":1,"thread_start_us":10101,"threads_started":1}
I20260619 16:56:49.757328 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.736451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.16.48.1:44989)
0619 16:56:49.745723 (+ 9272us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:49.745733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:49.746306 (+ 573us) server_negotiation.cc:407] Connection header received
0619 16:56:49.746468 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:49.746476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:49.746768 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:49.746962 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:49.748011 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.749661 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.752253 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.753787 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.753925 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.754170 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:49.756797 (+ 2627us) server_negotiation.cc:299] Negotiation successful
0619 16:56:49.756985 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9089,"thread_start_us":8927,"threads_started":1}
I20260619 16:56:49.759071 27185 tablet_service.cc:1467] Processing CreateTablet for tablet 1b5f208b03df4f1dbc150625f17ffb3f (DEFAULT_TABLE table=test-table [id=47efc9e507b94a55b02328159c28e9e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:49.760634 27185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b5f208b03df4f1dbc150625f17ffb3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:49.763235 27051 tablet_service.cc:1467] Processing CreateTablet for tablet 1b5f208b03df4f1dbc150625f17ffb3f (DEFAULT_TABLE table=test-table [id=47efc9e507b94a55b02328159c28e9e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:49.764535 27051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b5f208b03df4f1dbc150625f17ffb3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:49.766443 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.746634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:41207 (local address 127.0.0.1:33872)
0619 16:56:49.755010 (+ 8376us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:49.755043 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:49.755143 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:49.759386 (+ 4243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:49.759391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:49.759421 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:49.759632 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.759648 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.761827 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.761832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.763685 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.763705 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.765626 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.765633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.765822 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.765869 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 16:56:49.765978 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:56:49.766139 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8281,"thread_start_us":11270,"threads_started":1}
I20260619 16:56:49.766595 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.746692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.16.48.3:41207)
0619 16:56:49.758846 (+ 12154us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:49.758851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:49.758880 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:56:49.758963 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:49.758969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:49.759168 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:49.759401 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:49.759822 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.761629 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.763899 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.765429 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.765583 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.765802 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:49.766050 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 16:56:49.766226 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11987,"thread_start_us":11821,"threads_started":1}
I20260619 16:56:49.770954 27319 tablet_service.cc:1467] Processing CreateTablet for tablet 1b5f208b03df4f1dbc150625f17ffb3f (DEFAULT_TABLE table=test-table [id=47efc9e507b94a55b02328159c28e9e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:56:49.772531 27319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b5f208b03df4f1dbc150625f17ffb3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:56:49.774049 27403 tablet_bootstrap.cc:492] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: Bootstrap starting.
I20260619 16:56:49.777742 27403 tablet_bootstrap.cc:654] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:49.778980 27403 log.cc:824] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:49.779392 27404 tablet_bootstrap.cc:492] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: Bootstrap starting.
I20260619 16:56:49.782609 27403 tablet_bootstrap.cc:492] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: No bootstrap required, opened a new log
I20260619 16:56:49.782907 27403 ts_tablet_manager.cc:1397] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 16:56:49.783176 27404 tablet_bootstrap.cc:654] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:49.784531 27404 log.cc:824] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:49.787024 27407 tablet_bootstrap.cc:492] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: Bootstrap starting.
I20260619 16:56:49.789255 27404 tablet_bootstrap.cc:492] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: No bootstrap required, opened a new log
I20260619 16:56:49.789559 27404 ts_tablet_manager.cc:1397] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 16:56:49.790597 27407 tablet_bootstrap.cc:654] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: Neither blocks nor log segments found. Creating new log.
I20260619 16:56:49.791638 27407 log.cc:824] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: Log is configured to *not* fsync() on all Append() calls
I20260619 16:56:49.793435 27403 raft_consensus.cc:348] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } }
I20260619 16:56:49.793967 27403 raft_consensus.cc:374] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:49.794118 27403 raft_consensus.cc:729] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d22fe11a0bb412c956ff8d5282dd15c, State: Initialized, Role: FOLLOWER
I20260619 16:56:49.794831 27407 tablet_bootstrap.cc:492] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: No bootstrap required, opened a new log
I20260619 16:56:49.794811 27403 consensus_queue.cc:260] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } }
I20260619 16:56:49.795166 27407 ts_tablet_manager.cc:1397] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 16:56:49.796016 27250 heartbeater.cc:503] Master 127.16.48.62:45255 was elected leader, sending a full tablet report...
I20260619 16:56:49.798302 27403 ts_tablet_manager.cc:1428] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260619 16:56:49.799793 27404 raft_consensus.cc:348] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } }
I20260619 16:56:49.800242 27404 raft_consensus.cc:374] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:49.800376 27404 raft_consensus.cc:729] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a254f865a549fb8b2d388d3e5a1190, State: Initialized, Role: FOLLOWER
I20260619 16:56:49.801091 27404 consensus_queue.cc:260] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } }
I20260619 16:56:49.803892 27404 ts_tablet_manager.cc:1428] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 16:56:49.805399 27407 raft_consensus.cc:348] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } }
I20260619 16:56:49.805938 27407 raft_consensus.cc:374] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:56:49.806077 27407 raft_consensus.cc:729] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f941aa85a3b74525ba6b7a3bbfd18850, State: Initialized, Role: FOLLOWER
I20260619 16:56:49.806699 27407 consensus_queue.cc:260] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } }
I20260619 16:56:49.807873 27384 heartbeater.cc:503] Master 127.16.48.62:45255 was elected leader, sending a full tablet report...
I20260619 16:56:49.808570 27407 ts_tablet_manager.cc:1428] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
W20260619 16:56:49.834708 27117 tablet.cc:2369] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:56:49.877624 27251 tablet.cc:2369] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:56:49.909792 27385 tablet.cc:2369] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:56:49.931027 27411 raft_consensus.cc:482] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:56:49.931377 27411 raft_consensus.cc:504] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } }
I20260619 16:56:49.933197 27411 leader_election.cc:288] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d22fe11a0bb412c956ff8d5282dd15c (127.16.48.2:44987), 02a254f865a549fb8b2d388d3e5a1190 (127.16.48.1:44989)
I20260619 16:56:49.939394 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.933485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:44987 (local address 127.16.48.3:41177)
0619 16:56:49.933764 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:49.933792 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:49.933934 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:49.934338 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:49.934347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:49.934381 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:49.934591 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.934600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.935999 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.936007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.937482 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.937494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.938623 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.938628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.938808 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.938833 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:56:49.938942 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:56:49.939078 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":26}
I20260619 16:56:49.939369 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.933511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:41177 (local address 127.16.48.2:44987)
0619 16:56:49.933974 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:49.933982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:49.934004 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:56:49.934080 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:49.934087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:49.934177 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:49.934281 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:49.934771 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.935874 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.937666 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.938441 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.938603 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.938699 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:49.938972 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 16:56:49.939102 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321}
I20260619 16:56:49.940245 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.933841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:32787 (local address 127.16.48.1:44989)
0619 16:56:49.934089 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:49.934098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:49.934889 (+ 791us) server_negotiation.cc:407] Connection header received
0619 16:56:49.935055 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:49.935058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:49.935117 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:49.935182 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:56:49.935812 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.936717 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.938423 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.939213 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.939387 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.939490 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:49.939841 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 16:56:49.939980 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:56:49.940373 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.933641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:44989 (local address 127.16.48.3:32787)
0619 16:56:49.934768 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0619 16:56:49.934788 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:56:49.934924 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:56:49.935230 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:56:49.935235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:56:49.935260 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:56:49.935633 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.935644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.936881 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.936886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.938257 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:56:49.938269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.939443 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.939449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:56:49.939649 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.939695 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 16:56:49.939813 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 16:56:49.939947 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":1,"thread_start_us":560,"threads_started":1}
I20260619 16:56:49.940560 27205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5f208b03df4f1dbc150625f17ffb3f" candidate_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" is_pre_election: true
I20260619 16:56:49.941151 27205 raft_consensus.cc:2454] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f941aa85a3b74525ba6b7a3bbfd18850 in term 0.
I20260619 16:56:49.941568 27071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5f208b03df4f1dbc150625f17ffb3f" candidate_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a254f865a549fb8b2d388d3e5a1190" is_pre_election: true
I20260619 16:56:49.941928 27273 leader_election.cc:302] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d22fe11a0bb412c956ff8d5282dd15c, f941aa85a3b74525ba6b7a3bbfd18850; no voters:
I20260619 16:56:49.942162 27071 raft_consensus.cc:2454] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f941aa85a3b74525ba6b7a3bbfd18850 in term 0.
I20260619 16:56:49.942390 27411 raft_consensus.cc:2781] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:56:49.942546 27411 raft_consensus.cc:482] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:56:49.942687 27411 raft_consensus.cc:3037] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:49.945052 27411 raft_consensus.cc:504] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } }
I20260619 16:56:49.945689 27411 leader_election.cc:288] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [CANDIDATE]: Term 1 election: Requested vote from peers 3d22fe11a0bb412c956ff8d5282dd15c (127.16.48.2:44987), 02a254f865a549fb8b2d388d3e5a1190 (127.16.48.1:44989)
I20260619 16:56:49.946256 27071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5f208b03df4f1dbc150625f17ffb3f" candidate_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a254f865a549fb8b2d388d3e5a1190"
I20260619 16:56:49.946256 27205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5f208b03df4f1dbc150625f17ffb3f" candidate_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d22fe11a0bb412c956ff8d5282dd15c"
I20260619 16:56:49.946485 27205 raft_consensus.cc:3037] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:49.946496 27071 raft_consensus.cc:3037] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:56:49.949975 27205 raft_consensus.cc:2454] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f941aa85a3b74525ba6b7a3bbfd18850 in term 1.
I20260619 16:56:49.949976 27071 raft_consensus.cc:2454] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f941aa85a3b74525ba6b7a3bbfd18850 in term 1.
I20260619 16:56:49.950512 27274 leader_election.cc:302] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a254f865a549fb8b2d388d3e5a1190, f941aa85a3b74525ba6b7a3bbfd18850; no voters:
I20260619 16:56:49.950950 27411 raft_consensus.cc:2781] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:56:49.951431 27411 raft_consensus.cc:686] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 1 LEADER]: Becoming Leader. State: Replica: f941aa85a3b74525ba6b7a3bbfd18850, State: Running, Role: LEADER
I20260619 16:56:49.951936 27411 consensus_queue.cc:237] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } }
I20260619 16:56:49.956681 26927 catalog_manager.cc:5495] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 reported cstate change: term changed from 0 to 1, leader changed from <none> to f941aa85a3b74525ba6b7a3bbfd18850 (127.16.48.3). New cstate: current_term: 1 leader_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f941aa85a3b74525ba6b7a3bbfd18850" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 41207 } health_report { overall_health: HEALTHY } } }
I20260619 16:56:49.988785 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:49.983074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.16.48.3:41207)
0619 16:56:49.983333 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:49.983341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:49.983398 (+ 57us) server_negotiation.cc:407] Connection header received
0619 16:56:49.983624 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:49.983629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:49.983711 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:49.983849 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:56:49.984234 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.984971 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.986395 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:49.986828 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:49.986936 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:49.987484 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:56:49.988027 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:56:49.988236 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:49.988422 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 16:56:49.988557 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 16:56:49.997656 27205 raft_consensus.cc:1261] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [term 1 FOLLOWER]: Refusing update from remote peer f941aa85a3b74525ba6b7a3bbfd18850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:49.997660 27071 raft_consensus.cc:1261] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [term 1 FOLLOWER]: Refusing update from remote peer f941aa85a3b74525ba6b7a3bbfd18850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:56:49.998726 27413 consensus_queue.cc:1035] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02a254f865a549fb8b2d388d3e5a1190" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:49.999097 27411 consensus_queue.cc:1035] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d22fe11a0bb412c956ff8d5282dd15c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:56:50.011150 27416 mvcc.cc:205] Tried to move back new op lower bound from 7298614108139278336 to 7298614107967201280. Current Snapshot: MvccSnapshot[applied={T|T < 7298614108139278336}]
I20260619 16:56:50.012902 27418 mvcc.cc:205] Tried to move back new op lower bound from 7298614108139278336 to 7298614107967201280. Current Snapshot: MvccSnapshot[applied={T|T < 7298614108139278336}]
I20260619 16:56:50.014492 27417 mvcc.cc:205] Tried to move back new op lower bound from 7298614108139278336 to 7298614107967201280. Current Snapshot: MvccSnapshot[applied={T|T < 7298614108139278336}]
I20260619 16:56:50.025489 26926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:50.025830 26926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:56:50.029969 26926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f71c52a7d7ff412fbcd109fbdb05d4c0: Sending DeleteTablet for 3 replicas of tablet 1b5f208b03df4f1dbc150625f17ffb3f
I20260619 16:56:50.031255 27185 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b5f208b03df4f1dbc150625f17ffb3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:50 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:59890
I20260619 16:56:50.031399 27051 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b5f208b03df4f1dbc150625f17ffb3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:50 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:56196
I20260619 16:56:50.032639 27319 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b5f208b03df4f1dbc150625f17ffb3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:56:50 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:33872
I20260619 16:56:50.033470 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 26988
I20260619 16:56:50.037449 27427 tablet_replica.cc:330] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: stopping tablet replica
I20260619 16:56:50.040887 27427 raft_consensus.cc:2229] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:56:50.042039 27427 raft_consensus.cc:2258] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:50.050630 27427 ts_tablet_manager.cc:1905] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:50.052814 27425 tablet_replica.cc:330] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: stopping tablet replica
Jun 19 16:56:50 dist-test-slave-3ffd krb5kdc[26882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:56:50.053581 27426 tablet_replica.cc:330] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: stopping tablet replica
I20260619 16:56:50.053989 27426 raft_consensus.cc:2229] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:50.054523 27426 raft_consensus.cc:2258] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:50.054963 27425 raft_consensus.cc:2229] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:56:50.055505 27425 raft_consensus.cc:2258] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:56:50.057358 27426 ts_tablet_manager.cc:1905] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:50.059800 27427 ts_tablet_manager.cc:1918] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:50.060194 27427 log.cc:1196] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/1b5f208b03df4f1dbc150625f17ffb3f
I20260619 16:56:50.060320 27425 ts_tablet_manager.cc:1905] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:56:50.060972 27427 ts_tablet_manager.cc:1939] T 1b5f208b03df4f1dbc150625f17ffb3f P f941aa85a3b74525ba6b7a3bbfd18850: Deleting consensus metadata
I20260619 16:56:50.063069 26913 catalog_manager.cc:4841] TS f941aa85a3b74525ba6b7a3bbfd18850 (127.16.48.3:41207): tablet 1b5f208b03df4f1dbc150625f17ffb3f (table test-table [id=47efc9e507b94a55b02328159c28e9e2]) successfully deleted
I20260619 16:56:50.065232 27426 ts_tablet_manager.cc:1918] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:50.065528 27426 log.cc:1196] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/1b5f208b03df4f1dbc150625f17ffb3f
I20260619 16:56:50.066154 27426 ts_tablet_manager.cc:1939] T 1b5f208b03df4f1dbc150625f17ffb3f P 02a254f865a549fb8b2d388d3e5a1190: Deleting consensus metadata
I20260619 16:56:50.067492 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:50.035014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.16.48.1:44989)
0619 16:56:50.035252 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:50.035261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:50.036753 (+ 1492us) server_negotiation.cc:407] Connection header received
0619 16:56:50.038066 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:50.038076 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:50.038231 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:50.038340 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:50.043344 (+ 5004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:50.044918 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:50.048110 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:50.048873 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:50.048994 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:50.054588 (+ 5594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:50.054604 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:50.054628 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:50.054681 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:50.058662 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:50.063790 (+ 5128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:50.063801 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:50.063805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:50.064027 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:50.064675 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:50.064681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:50.064685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:50.065217 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:50.065537 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:50.066800 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0619 16:56:50.067064 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:56:50.068392 27111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:50.068568 27425 ts_tablet_manager.cc:1918] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:56:50.068809 27425 log.cc:1196] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/1b5f208b03df4f1dbc150625f17ffb3f
I20260619 16:56:50.068967 26913 catalog_manager.cc:4841] TS 02a254f865a549fb8b2d388d3e5a1190 (127.16.48.1:44989): tablet 1b5f208b03df4f1dbc150625f17ffb3f (table test-table [id=47efc9e507b94a55b02328159c28e9e2]) successfully deleted
I20260619 16:56:50.069463 27425 ts_tablet_manager.cc:1939] T 1b5f208b03df4f1dbc150625f17ffb3f P 3d22fe11a0bb412c956ff8d5282dd15c: Deleting consensus metadata
I20260619 16:56:50.072232 26912 catalog_manager.cc:4841] TS 3d22fe11a0bb412c956ff8d5282dd15c (127.16.48.2:44987): tablet 1b5f208b03df4f1dbc150625f17ffb3f (table test-table [id=47efc9e507b94a55b02328159c28e9e2]) successfully deleted
I20260619 16:56:50.170632 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 26988
I20260619 16:56:50.188107 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 27122
Jun 19 16:56:50 dist-test-slave-3ffd krb5kdc[26882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:56:50.205237 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:50.188997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.16.48.2:44987)
0619 16:56:50.189174 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:50.189181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:50.189200 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:56:50.189361 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:50.189367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:50.189464 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:50.189574 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:50.190855 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:50.191638 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:50.193064 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:50.193482 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:50.193631 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:50.197927 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:50.197948 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:50.197972 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:50.198031 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:50.202039 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:50.202867 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:50.202878 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:50.202882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:50.203049 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:50.203428 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:50.203433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:50.203436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:50.203921 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:50.204378 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:50.204582 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 16:56:50.204820 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:56:50.206171 27245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:50.295845 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 27122
I20260619 16:56:50.314320 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 27256
Jun 19 16:56:50 dist-test-slave-3ffd krb5kdc[26882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:56:50.331120 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:56:50.315132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.16.48.3:41207)
0619 16:56:50.315337 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:56:50.315342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:56:50.315361 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:56:50.315523 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:56:50.315528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:56:50.315623 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:56:50.315753 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:56:50.317062 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:50.317963 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:50.319303 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:56:50.319788 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:56:50.319939 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:56:50.324804 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:56:50.324817 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:56:50.324836 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:56:50.324879 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:56:50.328050 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:50.328811 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:50.328828 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:50.328831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:50.329015 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:56:50.329452 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:56:50.329457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:56:50.329459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:56:50.329814 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:56:50.330139 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 16:56:50.330488 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 16:56:50.330737 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:56:50.331858 27379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:50.453598 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 27256
I20260619 16:56:50.473524 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 26895
I20260619 16:56:50.474745 26957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:56:50.582909 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 26895
2026-06-19T16:56:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4536 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:56:50 dist-test-slave-3ffd krb5kdc[27435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:56:50 dist-test-slave-3ffd krb5kdc[27435](info): set up 2 sockets
Jun 19 16:56:50 dist-test-slave-3ffd krb5kdc[27435](info): commencing operation
krb5kdc: starting...
W20260619 16:56:51.709353 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:56:51 dist-test-slave-3ffd krb5kdc[27435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:56:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/rangeradmin_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/rangeradmin_127.16.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/rangerlookup_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/rangerlookup_127.16.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/HTTP_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/HTTP_127.16.48.20@KRBTEST.COM.keytab.
I20260619 16:56:52.071818 16576 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbQ4sJq/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/postgres -l logfile start
2026-06-19 16:56:54.772 UTC [27465] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:56:54.772 UTC [27465] LOG: listening on IPv4 address "127.16.48.20", port 54523
2026-06-19 16:56:54.776 UTC [27465] LOG: listening on Unix socket "/tmp/.s.PGSQL.54523"
2026-06-19 16:56:54.802 UTC [27468] LOG: database system was shut down at 2026-06-19 16:56:53 UTC
2026-06-19 16:56:54.806 UTC [27465] LOG: database system is ready to accept connections
I20260619 16:56:55.777103 16576 mini_postgres.cc:96] Postgres bound to 54523
2026-06-19 16:56:55.795 UTC [27476] FATAL: database "slave" does not exist
127.16.48.20:54523 - accepting connections
I20260619 16:56:55.796406 16576 mini_ranger.cc:161] Starting Ranger...
I20260619 16:56:55.825163 16576 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:56:56.355129 16576 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:56:56.355249 16576 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-admin
2026-06-19 16:56:56,705 [I] DB FLAVOR :POSTGRES
2026-06-19 16:56:56,706 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:56:56,706 [I] Checking connection..
2026-06-19 16:56:56,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:56:56,987 [I] Checking connection passed.
2026-06-19 16:56:56,988 [I] --------- Verifying version history table ---------
2026-06-19 16:56:56,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:56:57,234 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:56:57,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:56:57,498 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:56:57,498 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:56:57,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:56:57.748 UTC [27540] WARNING: there is no transaction in progress
2026-06-19 16:56:57,755 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:56:57,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:56:58,000 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:56:58,000 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:56:58,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:56:58,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:56:58,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3ffd.c.gcp-upstream.internal','N') ;"
2026-06-19 16:56:58,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:56:59,038 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:56:59,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:56:59,304 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:56:59,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:56:59,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:56:59,822 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:56:59,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:57:00.920 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:00.941 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:00.959 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:00.976 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:00.998 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.019 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.083 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.093 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.109 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.121 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.133 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.147 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.158 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.166 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.174 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01.182 UTC [27648] WARNING: there is no transaction in progress
2026-06-19 16:57:01,956 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:57:01,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:57:02,226 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:57:02,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:57:02,498 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:57:02,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:57:02,770 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:57:02,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:57:03,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:57:03,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3ffd.c.gcp-upstream.internal';"
2026-06-19 16:57:03,529 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:57:03,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:57:03,793 [I] DB_PATCHES have already been applied
I20260619 16:57:03.799688 16576 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbQ4sJq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:57:03.799820 16576 mini_ranger.cc:193] Using host: 127.16.48.20
I20260619 16:57:03.816255 16576 mini_ranger.cc:227] Ranger admin URL: http://127.16.48.20:32915
Jun 19, 2026 4:57:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:57:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:57:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:57:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:57:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:57:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:57:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:57:06.132477 16576 mini_ranger.cc:160] Time spent starting Ranger: real 10.336s user 0.016s sys 0.041s
Jun 19, 2026 4:57:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:57:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 4:57:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:57:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:57:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:57:28.272166 16576 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/rangerkms_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/rangerkms_127.16.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.48.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.48.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.48.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/HTTP_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.48.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/HTTP_127.16.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:57:28 dist-test-slave-3ffd krb5kdc[27435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888248, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:57:28.470374 16576 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:57:28.493357 16576 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:57:29.494396 16576 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:57:29.494566 16576 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-kms
I20260619 16:57:29.494731 16576 mini_ranger_kms.cc:227] Using postgres at 127.16.48.20:54523
2026-06-19 16:57:29,915 [I] DB FLAVOR :POSTGRES
2026-06-19 16:57:29,916 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:57:29,916 [I] Checking connection
2026-06-19 16:57:29,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:57:30,190 [I] connection success
2026-06-19 16:57:30,190 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:57:30,190 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:57:30,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:57:30,457 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:57:30,457 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:57:30,458 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:57:30,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:54523/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:57:30,773 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:57:31.278241 16576 mini_ranger_kms.cc:310] Created kms service
I20260619 16:57:31.523938 16576 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:57:31.573012 16576 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:57:31 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888248, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:57:32.005106 16576 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:57:32.005450 16576 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:57:32.005491 16576 mini_ranger_kms.cc:240] Using host: 127.16.48.20
I20260619 16:57:32.028828 16576 mini_ranger_kms.cc:276] Ranger KMS PID: 27830
I20260619 16:57:32.029028 16576 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.48.20:44307
Jun 19, 2026 4:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:57:34.332710 16576 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.862s user 0.127s sys 0.308s
I20260619 16:57:34.332973 16576 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:57:34.333189 16576 mini_ranger_kms.cc:401] 127.16.48.20:44307/kms/v1/keys
Jun 19 16:57:34 dist-test-slave-3ffd krb5kdc[27435](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781888254, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.48.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:57:34 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888254, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.48.20@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
Jun 19 16:57:35 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888254, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.48.20@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
Jun 19 16:57:37 dist-test-slave-3ffd krb5kdc[27435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:57:37.691704 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:43087
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:43087
--ranger_config_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:44307/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:57:38.067981 27880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:57:38.068346 27880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:57:38.068506 27880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:57:38.076736 27880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:57:38.076874 27880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:57:38.076961 27880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:57:38.077051 27880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:57:38.077147 27880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:57:38.087903 27880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.48.20:44307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:43087
--ranger_config_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:43087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:57:38.089363 27880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:57:38.090827 27880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:57:38.104777 27888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:57:38.104861 27885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:57:38.104805 27886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:57:38.105310 27880 server_base.cc:1034] running on GCE node
I20260619 16:57:38.106328 27880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:57:38.108250 27880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:57:38.109529 27880 hybrid_clock.cc:648] HybridClock initialized: now 1781888258109485 us; error 50 us; skew 500 ppm
I20260619 16:57:38.110002 27880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:57:38 dist-test-slave-3ffd krb5kdc[27435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888258, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:57:38.113633 27880 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:57:38.115703 27880 webserver.cc:460] Webserver started at http://127.16.48.62:33309/ using document root <none> and password file <none>
I20260619 16:57:38.116535 27880 fs_manager.cc:359] Metadata directory not provided
I20260619 16:57:38.116680 27880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:57:38.117015 27880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:57:38 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888258, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:57:38.186512 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "d6950699d8e543cc95d7d8a1e7a7dde1"
format_stamp: "Formatted at 2026-06-19 16:57:38 on dist-test-slave-3ffd"
server_key: "a6f26eec5faa9e8abe29c857caaff1f4"
server_key_iv: "52596ec47509272d2f656bd715600f20"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:38.187803 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "d6950699d8e543cc95d7d8a1e7a7dde1"
format_stamp: "Formatted at 2026-06-19 16:57:38 on dist-test-slave-3ffd"
server_key: "a6f26eec5faa9e8abe29c857caaff1f4"
server_key_iv: "52596ec47509272d2f656bd715600f20"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:38.196353 27880 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260619 16:57:38.219393 27895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:57:38.221171 27880 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 16:57:38.221390 27880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "d6950699d8e543cc95d7d8a1e7a7dde1"
format_stamp: "Formatted at 2026-06-19 16:57:38 on dist-test-slave-3ffd"
server_key: "a6f26eec5faa9e8abe29c857caaff1f4"
server_key_iv: "52596ec47509272d2f656bd715600f20"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:38.221642 27880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:57:38.239042 27880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:57:38.243510 27880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:57:38.243868 27880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:57:38.265328 27880 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:43087
I20260619 16:57:38.265359 27946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:43087 every 8 connection(s)
I20260619 16:57:38.266882 27880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:57:38.271068 27947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:57:38.276209 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 27880
I20260619 16:57:38.276440 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:57:38.277117 16576 external_mini_cluster.cc:1428] Setting key 8cd844c67580b4a09403e27de085dbde
I20260619 16:57:38.282747 27947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1: Bootstrap starting.
I20260619 16:57:38.286727 27947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1: Neither blocks nor log segments found. Creating new log.
I20260619 16:57:38.288203 27947 log.cc:824] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:57:38.292150 27947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1: No bootstrap required, opened a new log
Jun 19 16:57:38 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:57:38.303835 27947 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6950699d8e543cc95d7d8a1e7a7dde1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43087 } }
I20260619 16:57:38.304378 27947 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:57:38.304494 27947 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6950699d8e543cc95d7d8a1e7a7dde1, State: Initialized, Role: FOLLOWER
I20260619 16:57:38.305132 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:38.279229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.16.48.62:43087)
0619 16:57:38.280213 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:38.280229 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:38.280278 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:57:38.282493 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:38.282526 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:38.283033 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:38.283432 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:57:38.285261 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:38.286645 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:38.288565 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:38.289780 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:38.289942 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:38.294972 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:57:38.294985 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:57:38.295014 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:57:38.295084 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:57:38.299829 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:38.300816 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:38.300827 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:38.300838 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:38.301059 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:38.301457 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:38.301462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:38.301466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:38.302351 (+ 885us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:57:38.302688 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:38.303142 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 16:57:38.303627 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":433,"threads_started":1}
I20260619 16:57:38.305270 27947 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6950699d8e543cc95d7d8a1e7a7dde1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43087 } }
I20260619 16:57:38.305603 27947 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:57:38.305799 27947 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:57:38.306000 27947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:57:38.309330 27947 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6950699d8e543cc95d7d8a1e7a7dde1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43087 } }
I20260619 16:57:38.309922 27947 leader_election.cc:302] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6950699d8e543cc95d7d8a1e7a7dde1; no voters:
I20260619 16:57:38.310576 27947 leader_election.cc:288] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:57:38.310741 27952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:57:38.311462 27952 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [term 1 LEADER]: Becoming Leader. State: Replica: d6950699d8e543cc95d7d8a1e7a7dde1, State: Running, Role: LEADER
I20260619 16:57:38.312286 27952 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6950699d8e543cc95d7d8a1e7a7dde1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43087 } }
I20260619 16:57:38.312829 27947 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:57:38.315830 27953 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6950699d8e543cc95d7d8a1e7a7dde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6950699d8e543cc95d7d8a1e7a7dde1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43087 } } }
I20260619 16:57:38.316355 27953 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:57:38.316895 27954 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6950699d8e543cc95d7d8a1e7a7dde1. Latest consensus state: current_term: 1 leader_uuid: "d6950699d8e543cc95d7d8a1e7a7dde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6950699d8e543cc95d7d8a1e7a7dde1" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43087 } } }
I20260619 16:57:38.317210 27954 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:57:38.317240 27961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:57:38.324419 27961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:57:38.325455 27947 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:57:38.344728 27961 catalog_manager.cc:1347] Generated new cluster ID: c40695234db64bafbf3667de642970a3
I20260619 16:57:38.344904 27961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:57:38.362708 27961 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:57:38.364292 27961 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:57:38.389254 27961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1: Generated new TSK 0
I20260619 16:57:38.390528 27961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:57:39.184618 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:44307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43087
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:57:39 dist-test-slave-3ffd krb5kdc[27435](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781888259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:57:39.632685 27993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:57:39.633059 27993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:57:39.633229 27993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:57:39.643206 27993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:57:39.643384 27993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:57:39.643576 27993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:57:39.655993 27993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.48.20:44307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:57:39.657655 27993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:57:39.659694 27993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:57:39.678889 28003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:57:39.679863 28004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:57:39.684829 28006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:57:39.685590 27993 server_base.cc:1034] running on GCE node
I20260619 16:57:39.686407 27993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:57:39.687590 27993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:57:39.688850 27993 hybrid_clock.cc:648] HybridClock initialized: now 1781888259688785 us; error 71 us; skew 500 ppm
I20260619 16:57:39.689218 27993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:57:39 dist-test-slave-3ffd krb5kdc[27435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:57:39.692652 27993 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:57:39.697125 27993 webserver.cc:460] Webserver started at http://127.16.48.1:39887/ using document root <none> and password file <none>
I20260619 16:57:39.697939 27993 fs_manager.cc:359] Metadata directory not provided
I20260619 16:57:39.698103 27993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:57:39.698462 27993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:57:39 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:57:39.730042 27993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "7e54f3fcf9d24e64ab4c49314eab813a"
format_stamp: "Formatted at 2026-06-19 16:57:39 on dist-test-slave-3ffd"
server_key: "0fa3086f387f5950ec864500d5a6c532"
server_key_iv: "62b9dee6ffea261708f6e42ccf1a648a"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:39.731703 27993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "7e54f3fcf9d24e64ab4c49314eab813a"
format_stamp: "Formatted at 2026-06-19 16:57:39 on dist-test-slave-3ffd"
server_key: "0fa3086f387f5950ec864500d5a6c532"
server_key_iv: "62b9dee6ffea261708f6e42ccf1a648a"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:39.739835 27993 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260619 16:57:39.768888 28013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:57:39.771034 27993 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260619 16:57:39.771242 27993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "7e54f3fcf9d24e64ab4c49314eab813a"
format_stamp: "Formatted at 2026-06-19 16:57:39 on dist-test-slave-3ffd"
server_key: "0fa3086f387f5950ec864500d5a6c532"
server_key_iv: "62b9dee6ffea261708f6e42ccf1a648a"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:39.771522 27993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:57:39.794829 27993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:57:39.800709 27993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:57:39.801133 27993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:57:39.808674 27993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:57:39.817158 27993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:57:39.817365 27993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 16:57:39.817585 27993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:57:39.817727 27993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:57:39.908488 27993 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:42339
I20260619 16:57:39.909180 28125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:42339 every 8 connection(s)
I20260619 16:57:39.910357 27993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:57:39.920876 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 27993
I20260619 16:57:39.921155 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:57:39.921984 16576 external_mini_cluster.cc:1428] Setting key 258922451255737ac6ac6f2aff8cef18
Jun 19 16:57:39 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:57:39.947880 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:39.918463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:60579 (local address 127.16.48.62:43087)
0619 16:57:39.922713 (+ 4250us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:39.922721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:39.923498 (+ 777us) server_negotiation.cc:407] Connection header received
0619 16:57:39.926607 (+ 3109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:39.926614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:39.926729 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:39.926814 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:57:39.929405 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:39.930695 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:39.933937 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:39.934607 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:39.934750 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:39.940095 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:57:39.940105 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:57:39.940113 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:57:39.940184 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:57:39.943871 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:39.945046 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:39.945051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:39.945054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:39.945210 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:39.945836 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:39.945848 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:39.945850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:39.946140 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:57:39.946308 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:39.947220 (+ 912us) server_negotiation.cc:299] Negotiation successful
0619 16:57:39.947469 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4103,"thread_start_us":3983,"threads_started":1}
I20260619 16:57:39.949239 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:39.918965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43087 (local address 127.16.48.1:60579)
0619 16:57:39.923311 (+ 4346us) negotiation.cc:107] Waiting for socket to connect
0619 16:57:39.923370 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:57:39.926111 (+ 2741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:57:39.927435 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:57:39.927454 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:57:39.928177 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:57:39.929158 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:39.929192 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:39.930941 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:39.930948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:39.933704 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:39.933725 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:39.935153 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:39.935163 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:39.935570 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:39.936565 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:57:39.936607 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:57:39.939752 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:57:39.944073 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:57:39.944088 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:57:39.944113 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:57:39.944889 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:57:39.945347 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:57:39.945355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:57:39.945359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:57:39.945709 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:57:39.946287 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:57:39.946296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:57:39.946800 (+ 504us) client_negotiation.cc:769] Sending connection context
0619 16:57:39.947127 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 16:57:39.947729 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3994,"thread_start_us":7753,"threads_started":1}
I20260619 16:57:39.951107 28126 heartbeater.cc:347] Connected to a master server at 127.16.48.62:43087
I20260619 16:57:39.951463 28126 heartbeater.cc:464] Registering TS with master...
I20260619 16:57:39.952498 28126 heartbeater.cc:511] Master 127.16.48.62:43087 requested a full tablet report, sending...
I20260619 16:57:39.955102 27912 ts_manager.cc:194] Registered new tserver with Master: 7e54f3fcf9d24e64ab4c49314eab813a (127.16.48.1:42339)
I20260619 16:57:39.956758 27912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:60579
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
Jun 19 16:57:40 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:57:40.008739 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:44307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43087
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:57:40 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
W20260619 16:57:40.555516 28132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:57:40.556082 28132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:57:40.556351 28132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:57:40.568897 28132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:57:40.569172 28132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:57:40.569392 28132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:57:40.584941 28132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.48.20:44307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:57:40.587030 28132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:57:40.589010 28132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:57:40.624392 28139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:57:40.626197 28140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:57:40.630448 28142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:57:40.638188 28132 server_base.cc:1034] running on GCE node
I20260619 16:57:40.639394 28132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:57:40.648527 28132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:57:40.656229 28132 hybrid_clock.cc:648] HybridClock initialized: now 1781888260655316 us; error 1113 us; skew 500 ppm
I20260619 16:57:40.656769 28132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:57:40 dist-test-slave-3ffd krb5kdc[27435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888260, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:57:40.661612 28132 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:57:40.688459 28132 webserver.cc:460] Webserver started at http://127.16.48.2:45879/ using document root <none> and password file <none>
I20260619 16:57:40.689656 28132 fs_manager.cc:359] Metadata directory not provided
I20260619 16:57:40.689841 28132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:57:40.690194 28132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:57:40 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888260, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:57:40.724929 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "07de2c6a688e417b82a919f782b1ed7b"
format_stamp: "Formatted at 2026-06-19 16:57:40 on dist-test-slave-3ffd"
server_key: "ea1e2cadb8b6daceb7f6be0c21f30931"
server_key_iv: "2cacbff9f94181f8eeabc1ef4fce5d1e"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:40.726758 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "07de2c6a688e417b82a919f782b1ed7b"
format_stamp: "Formatted at 2026-06-19 16:57:40 on dist-test-slave-3ffd"
server_key: "ea1e2cadb8b6daceb7f6be0c21f30931"
server_key_iv: "2cacbff9f94181f8eeabc1ef4fce5d1e"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:40.740741 28132 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.012s sys 0.001s
I20260619 16:57:40.784862 28156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:57:40.787250 28132 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.012s
I20260619 16:57:40.787675 28132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "07de2c6a688e417b82a919f782b1ed7b"
format_stamp: "Formatted at 2026-06-19 16:57:40 on dist-test-slave-3ffd"
server_key: "ea1e2cadb8b6daceb7f6be0c21f30931"
server_key_iv: "2cacbff9f94181f8eeabc1ef4fce5d1e"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:40.788056 28132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:57:40.846753 28132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:57:40.852391 28132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:57:40.852787 28132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:57:40.860387 28132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:57:40.868911 28132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:57:40.869130 28132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 16:57:40.869302 28132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:57:40.869391 28132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:57:40.961038 28126 heartbeater.cc:503] Master 127.16.48.62:43087 was elected leader, sending a full tablet report...
I20260619 16:57:41.003372 28132 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:42269
I20260619 16:57:41.003454 28268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:42269 every 8 connection(s)
I20260619 16:57:41.005267 28132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:57:41.014706 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 28132
I20260619 16:57:41.014889 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:57:41.015357 16576 external_mini_cluster.cc:1428] Setting key c0340687929cf0e49ddc94260bd9231b
Jun 19 16:57:41 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888260, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:57:41.049413 28271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.008402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:35063 (local address 127.16.48.62:43087)
0619 16:57:41.024398 (+ 15996us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.024406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.024432 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:57:41.027353 (+ 2921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.027359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.027442 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.027533 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:57:41.029825 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.030882 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.033478 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.034041 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.034163 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.041392 (+ 7229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:57:41.041400 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:57:41.041405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:57:41.041452 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:57:41.045431 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:41.046539 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:41.046545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:41.046548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:41.046704 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:41.047296 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:41.047301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:41.047304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:41.047642 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:57:41.047787 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:41.048806 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0619 16:57:41.049056 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15846,"thread_start_us":16042,"threads_started":1}
I20260619 16:57:41.051973 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.009118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43087 (local address 127.16.48.2:35063)
0619 16:57:41.022494 (+ 13376us) negotiation.cc:107] Waiting for socket to connect
0619 16:57:41.022570 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 16:57:41.026823 (+ 4253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:57:41.027971 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:57:41.027984 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:57:41.028656 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:57:41.029589 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.029615 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.031102 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.031108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.033277 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.033295 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.034469 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.034475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.034763 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.035614 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:57:41.035646 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:57:41.041029 (+ 5383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:57:41.045695 (+ 4666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:57:41.045705 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:57:41.045727 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:57:41.046382 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:57:41.046846 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:57:41.046850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:57:41.046853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:57:41.047157 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:57:41.047878 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:57:41.047886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:57:41.048400 (+ 514us) client_negotiation.cc:769] Sending connection context
0619 16:57:41.049991 (+ 1591us) client_negotiation.cc:240] Negotiation successful
0619 16:57:41.050581 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12977,"thread_start_us":16413,"threads_started":1}
I20260619 16:57:41.054016 28269 heartbeater.cc:347] Connected to a master server at 127.16.48.62:43087
I20260619 16:57:41.054347 28269 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
I20260619 16:57:41.055193 28269 heartbeater.cc:511] Master 127.16.48.62:43087 requested a full tablet report, sending...
I20260619 16:57:41.057138 27912 ts_manager.cc:194] Registered new tserver with Master: 07de2c6a688e417b82a919f782b1ed7b (127.16.48.2:42269)
I20260619 16:57:41.058240 27912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:35063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:57:41.095201 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:44307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43087
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:57:41.532531 28275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:57:41.532829 28275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:57:41.533018 28275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:57:41.541086 28275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:57:41.541215 28275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:57:41.541420 28275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:57:41.552661 28275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.48.20:44307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:57:41.554087 28275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:57:41.555485 28275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:57:41.569248 28280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:57:41.569566 28281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:57:41.570319 28283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:57:41.571296 28275 server_base.cc:1034] running on GCE node
I20260619 16:57:41.571966 28275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:57:41.573113 28275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:57:41.574342 28275 hybrid_clock.cc:648] HybridClock initialized: now 1781888261574282 us; error 40 us; skew 500 ppm
I20260619 16:57:41.574843 28275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:57:41 dist-test-slave-3ffd krb5kdc[27435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888261, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:57:41.578714 28275 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:57:41.580442 28275 webserver.cc:460] Webserver started at http://127.16.48.3:36269/ using document root <none> and password file <none>
I20260619 16:57:41.581178 28275 fs_manager.cc:359] Metadata directory not provided
I20260619 16:57:41.581281 28275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:57:41.581564 28275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:57:41 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888261, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:57:41.596699 28275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "8778b6c94b22457286f3eb6341261280"
format_stamp: "Formatted at 2026-06-19 16:57:41 on dist-test-slave-3ffd"
server_key: "9fe4c0bc7de4a55faa9f991da888752a"
server_key_iv: "bee3d22a3a53811fc8faad5d77ce72dc"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:41.597600 28275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "8778b6c94b22457286f3eb6341261280"
format_stamp: "Formatted at 2026-06-19 16:57:41 on dist-test-slave-3ffd"
server_key: "9fe4c0bc7de4a55faa9f991da888752a"
server_key_iv: "bee3d22a3a53811fc8faad5d77ce72dc"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:41.604039 28275 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 16:57:41.619940 28290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:57:41.621707 28275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:57:41.621934 28275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "8778b6c94b22457286f3eb6341261280"
format_stamp: "Formatted at 2026-06-19 16:57:41 on dist-test-slave-3ffd"
server_key: "9fe4c0bc7de4a55faa9f991da888752a"
server_key_iv: "bee3d22a3a53811fc8faad5d77ce72dc"
server_key_version: "kuduclusterkey@0"
I20260619 16:57:41.622166 28275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:57:41.637409 28275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:57:41.642185 28275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:57:41.642518 28275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:57:41.644018 28275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:57:41.645617 28275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:57:41.645694 28275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:57:41.645823 28275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:57:41.645881 28275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:57:41.683547 28275 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:37273
I20260619 16:57:41.683576 28402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:37273 every 8 connection(s)
I20260619 16:57:41.685143 28275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:57:41.695220 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 28275
I20260619 16:57:41.695436 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:57:41.695879 16576 external_mini_cluster.cc:1428] Setting key b5ceea9657ce8f7580b5b33782a25f00
Jun 19 16:57:41 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888261, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:57:41.706676 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.687908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:33451 (local address 127.16.48.62:43087)
0619 16:57:41.688410 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.688417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.689628 (+ 1211us) server_negotiation.cc:407] Connection header received
0619 16:57:41.691493 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.691498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.691589 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.691692 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:57:41.694109 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.694947 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.696922 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.697387 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.697529 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.700945 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:57:41.700950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:57:41.700954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:57:41.700988 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:57:41.703459 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:41.704324 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:41.704329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:41.704332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:41.704458 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:41.704896 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:41.704906 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:41.704908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:41.705204 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:57:41.705366 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:41.706076 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 16:57:41.706297 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":227,"threads_started":1}
I20260619 16:57:41.707672 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.688606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43087 (local address 127.16.48.3:33451)
0619 16:57:41.689412 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 16:57:41.689460 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:57:41.691185 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:57:41.691951 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:57:41.691960 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:57:41.692718 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:57:41.693909 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.693934 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.695122 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.695135 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.696760 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.696774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.697642 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.697650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.697900 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.698631 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:57:41.698655 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:57:41.700691 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:57:41.703621 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:57:41.703632 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:57:41.703650 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:57:41.704165 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:57:41.704574 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:57:41.704579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:57:41.704583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:57:41.704777 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:57:41.705336 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:57:41.705344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:57:41.705707 (+ 363us) client_negotiation.cc:769] Sending connection context
0619 16:57:41.705979 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 16:57:41.706524 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":354,"threads_started":1}
I20260619 16:57:41.709225 28403 heartbeater.cc:347] Connected to a master server at 127.16.48.62:43087
I20260619 16:57:41.709615 28403 heartbeater.cc:464] Registering TS with master...
I20260619 16:57:41.710528 28403 heartbeater.cc:511] Master 127.16.48.62:43087 requested a full tablet report, sending...
I20260619 16:57:41.712005 27912 ts_manager.cc:194] Registered new tserver with Master: 8778b6c94b22457286f3eb6341261280 (127.16.48.3:37273)
I20260619 16:57:41.712927 27912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:33451
I20260619 16:57:41.723959 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:57:41.743468 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.729074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.16.48.62:43087)
0619 16:57:41.729243 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.729248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.729764 (+ 516us) server_negotiation.cc:407] Connection header received
0619 16:57:41.730010 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.730017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.730098 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.730209 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:57:41.732215 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.732965 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.734541 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.734966 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.735111 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.737687 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:57:41.737692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:57:41.737695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:57:41.737732 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:57:41.740481 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:41.741253 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:41.741259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:41.741262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:41.741403 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:41.741821 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:41.741828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:41.741831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:41.742240 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:57:41.742511 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:41.742880 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 16:57:41.743130 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:57:41.747893 27912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:57:41.755604 27912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:57:41.793587 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.771751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:42269 (local address 127.0.0.1:49346)
0619 16:57:41.784145 (+ 12394us) negotiation.cc:107] Waiting for socket to connect
0619 16:57:41.784164 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:57:41.784315 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:57:41.785526 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:57:41.785532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:57:41.785566 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:57:41.786230 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.786251 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.788152 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.788156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.791014 (+ 2858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.791030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.792474 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.792481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.792683 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.792737 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:57:41.792900 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 16:57:41.793136 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12226,"mutex_wait_us":351,"thread_start_us":12073,"threads_started":1}
I20260619 16:57:41.794409 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.772449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.16.48.2:42269)
0619 16:57:41.783166 (+ 10717us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.783179 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.784487 (+ 1308us) server_negotiation.cc:407] Connection header received
0619 16:57:41.784632 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.784649 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.784949 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.785208 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:57:41.786443 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.787972 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.791216 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.792265 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.792412 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.792649 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:41.793857 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0619 16:57:41.794076 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10550,"thread_start_us":10422,"threads_started":1}
I20260619 16:57:41.794674 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.769391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:37273 (local address 127.0.0.1:37314)
0619 16:57:41.781979 (+ 12588us) negotiation.cc:107] Waiting for socket to connect
0619 16:57:41.782028 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:57:41.782244 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:57:41.785563 (+ 3319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:57:41.785567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:57:41.785587 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:57:41.786265 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.786280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.787747 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.787752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.790386 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.790405 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.793552 (+ 3147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.793562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.793832 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.793868 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:57:41.793967 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:57:41.794145 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12323,"mutex_wait_us":141,"thread_start_us":12185,"threads_started":1}
I20260619 16:57:41.795190 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.769161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42339 (local address 127.0.0.1:43398)
0619 16:57:41.782381 (+ 13220us) negotiation.cc:107] Waiting for socket to connect
0619 16:57:41.782417 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:57:41.782587 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:57:41.785314 (+ 2727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:57:41.785323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:57:41.785352 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:57:41.785951 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.785967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.788039 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.788048 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.791014 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.791030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.794153 (+ 3123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.794182 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.794388 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.794422 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:57:41.794600 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 16:57:41.794763 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12466,"mutex_wait_us":220,"thread_start_us":12784,"threads_started":1}
I20260619 16:57:41.795612 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.769994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.16.48.3:37273)
0619 16:57:41.784984 (+ 14990us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.784993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.785024 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:57:41.785112 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.785121 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.785349 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.785533 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:57:41.786505 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.787560 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.790576 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.792896 (+ 2320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.793041 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.793294 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:41.794982 (+ 1688us) server_negotiation.cc:299] Negotiation successful
0619 16:57:41.795180 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14808,"thread_start_us":15290,"threads_started":1}
I20260619 16:57:41.796099 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.769286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.16.48.1:42339)
0619 16:57:41.784602 (+ 15316us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.784615 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.784655 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:57:41.784773 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.784780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.785026 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.785919 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:57:41.786178 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.787847 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.791202 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.793199 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.793351 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.793972 (+ 621us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:41.795603 (+ 1631us) server_negotiation.cc:299] Negotiation successful
0619 16:57:41.795796 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15134,"thread_start_us":14983,"threads_started":1}
I20260619 16:57:41.798596 28204 tablet_service.cc:1467] Processing CreateTablet for tablet 6054766a06394fb4bbb2c0c8411b476c (DEFAULT_TABLE table=test-table [id=a341fdf0f70e4903b51df782d7025937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:57:41.800078 28204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6054766a06394fb4bbb2c0c8411b476c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:57:41.799675 28061 tablet_service.cc:1467] Processing CreateTablet for tablet 6054766a06394fb4bbb2c0c8411b476c (DEFAULT_TABLE table=test-table [id=a341fdf0f70e4903b51df782d7025937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:57:41.801311 28061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6054766a06394fb4bbb2c0c8411b476c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:57:41.801656 28338 tablet_service.cc:1467] Processing CreateTablet for tablet 6054766a06394fb4bbb2c0c8411b476c (DEFAULT_TABLE table=test-table [id=a341fdf0f70e4903b51df782d7025937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:57:41.803274 28338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6054766a06394fb4bbb2c0c8411b476c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:57:41.814014 28422 tablet_bootstrap.cc:492] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: Bootstrap starting.
I20260619 16:57:41.818153 28422 tablet_bootstrap.cc:654] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: Neither blocks nor log segments found. Creating new log.
I20260619 16:57:41.818642 28423 tablet_bootstrap.cc:492] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: Bootstrap starting.
I20260619 16:57:41.819702 28422 log.cc:824] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:57:41.820693 28424 tablet_bootstrap.cc:492] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: Bootstrap starting.
I20260619 16:57:41.823110 28423 tablet_bootstrap.cc:654] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: Neither blocks nor log segments found. Creating new log.
I20260619 16:57:41.824412 28423 log.cc:824] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:57:41.825103 28422 tablet_bootstrap.cc:492] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: No bootstrap required, opened a new log
I20260619 16:57:41.825497 28424 tablet_bootstrap.cc:654] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: Neither blocks nor log segments found. Creating new log.
I20260619 16:57:41.825616 28422 ts_tablet_manager.cc:1397] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260619 16:57:41.827039 28424 log.cc:824] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: Log is configured to *not* fsync() on all Append() calls
I20260619 16:57:41.828022 28423 tablet_bootstrap.cc:492] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: No bootstrap required, opened a new log
I20260619 16:57:41.828357 28423 ts_tablet_manager.cc:1397] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 16:57:41.830817 28424 tablet_bootstrap.cc:492] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: No bootstrap required, opened a new log
I20260619 16:57:41.831142 28424 ts_tablet_manager.cc:1397] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 16:57:41.836951 28423 raft_consensus.cc:348] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } }
I20260619 16:57:41.837435 28423 raft_consensus.cc:374] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:57:41.837543 28423 raft_consensus.cc:729] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e54f3fcf9d24e64ab4c49314eab813a, State: Initialized, Role: FOLLOWER
I20260619 16:57:41.838017 28422 raft_consensus.cc:348] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } }
I20260619 16:57:41.838253 28423 consensus_queue.cc:260] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } }
I20260619 16:57:41.838521 28422 raft_consensus.cc:374] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:57:41.838637 28422 raft_consensus.cc:729] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07de2c6a688e417b82a919f782b1ed7b, State: Initialized, Role: FOLLOWER
I20260619 16:57:41.839299 28422 consensus_queue.cc:260] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } }
I20260619 16:57:41.840780 28269 heartbeater.cc:503] Master 127.16.48.62:43087 was elected leader, sending a full tablet report...
I20260619 16:57:41.841560 28422 ts_tablet_manager.cc:1428] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260619 16:57:41.844005 28424 raft_consensus.cc:348] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } }
I20260619 16:57:41.844795 28424 raft_consensus.cc:374] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:57:41.844935 28424 raft_consensus.cc:729] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8778b6c94b22457286f3eb6341261280, State: Initialized, Role: FOLLOWER
I20260619 16:57:41.845914 28424 consensus_queue.cc:260] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } }
I20260619 16:57:41.850504 28428 raft_consensus.cc:482] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:57:41.850821 28428 raft_consensus.cc:504] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } }
I20260619 16:57:41.852280 28423 ts_tablet_manager.cc:1428] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: Time spent starting tablet: real 0.024s user 0.015s sys 0.007s
I20260619 16:57:41.853674 28403 heartbeater.cc:503] Master 127.16.48.62:43087 was elected leader, sending a full tablet report...
I20260619 16:57:41.855202 28424 ts_tablet_manager.cc:1428] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: Time spent starting tablet: real 0.024s user 0.016s sys 0.008s
I20260619 16:57:41.855526 28428 leader_election.cc:288] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8778b6c94b22457286f3eb6341261280 (127.16.48.3:37273), 07de2c6a688e417b82a919f782b1ed7b (127.16.48.2:42269)
I20260619 16:57:41.863642 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.856710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:56859 (local address 127.16.48.2:42269)
0619 16:57:41.856919 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.856927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.857911 (+ 984us) server_negotiation.cc:407] Connection header received
0619 16:57:41.858045 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.858049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.858123 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.858212 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:57:41.859072 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.859785 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.861770 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.862584 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.862759 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.862876 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:41.863220 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:57:41.863360 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:57:41.863878 28432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.853357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:37273 (local address 127.16.48.1:40667)
0619 16:57:41.857289 (+ 3932us) negotiation.cc:107] Waiting for socket to connect
0619 16:57:41.857314 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:57:41.857610 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:57:41.858068 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:57:41.858072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:57:41.858098 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:57:41.858467 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.858481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.860040 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.860048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.861654 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.861667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.862897 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.862904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.863113 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.863137 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:57:41.863248 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:57:41.863391 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3751,"mutex_wait_us":48,"thread_start_us":3601,"threads_started":1}
I20260619 16:57:41.863700 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.853486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:40667 (local address 127.16.48.3:37273)
0619 16:57:41.853715 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.853722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.857425 (+ 3703us) server_negotiation.cc:407] Connection header received
0619 16:57:41.857811 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.857819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.857919 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.858031 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:57:41.858787 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.859856 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.861822 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.862634 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.862807 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.862929 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:41.863275 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 16:57:41.863406 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"spinlock_wait_cycles":31104}
I20260619 16:57:41.864600 28433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.855818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:42269 (local address 127.16.48.1:56859)
0619 16:57:41.857767 (+ 1949us) negotiation.cc:107] Waiting for socket to connect
0619 16:57:41.857786 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:57:41.857924 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:57:41.858372 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:57:41.858379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:57:41.858407 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:57:41.858856 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.858869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.859949 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.859954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.861578 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:57:41.861593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.862854 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.862860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:57:41.863048 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.863073 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:57:41.863190 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:57:41.863341 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1801,"thread_start_us":1615,"threads_started":1}
I20260619 16:57:41.865116 28358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6054766a06394fb4bbb2c0c8411b476c" candidate_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8778b6c94b22457286f3eb6341261280" is_pre_election: true
I20260619 16:57:41.865706 28358 raft_consensus.cc:2454] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e54f3fcf9d24e64ab4c49314eab813a in term 0.
I20260619 16:57:41.865651 28224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6054766a06394fb4bbb2c0c8411b476c" candidate_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07de2c6a688e417b82a919f782b1ed7b" is_pre_election: true
I20260619 16:57:41.866539 28017 leader_election.cc:302] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e54f3fcf9d24e64ab4c49314eab813a, 8778b6c94b22457286f3eb6341261280; no voters:
I20260619 16:57:41.866626 28224 raft_consensus.cc:2454] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e54f3fcf9d24e64ab4c49314eab813a in term 0.
I20260619 16:57:41.866952 28428 raft_consensus.cc:2781] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:57:41.867117 28428 raft_consensus.cc:482] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:57:41.867252 28428 raft_consensus.cc:3037] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:57:41.869578 28428 raft_consensus.cc:504] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } }
I20260619 16:57:41.870270 28428 leader_election.cc:288] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [CANDIDATE]: Term 1 election: Requested vote from peers 8778b6c94b22457286f3eb6341261280 (127.16.48.3:37273), 07de2c6a688e417b82a919f782b1ed7b (127.16.48.2:42269)
I20260619 16:57:41.870774 28358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6054766a06394fb4bbb2c0c8411b476c" candidate_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8778b6c94b22457286f3eb6341261280"
I20260619 16:57:41.870857 28224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6054766a06394fb4bbb2c0c8411b476c" candidate_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07de2c6a688e417b82a919f782b1ed7b"
I20260619 16:57:41.871029 28358 raft_consensus.cc:3037] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:57:41.871045 28224 raft_consensus.cc:3037] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:57:41.874396 28358 raft_consensus.cc:2454] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e54f3fcf9d24e64ab4c49314eab813a in term 1.
I20260619 16:57:41.874396 28224 raft_consensus.cc:2454] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e54f3fcf9d24e64ab4c49314eab813a in term 1.
I20260619 16:57:41.874941 28015 leader_election.cc:302] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07de2c6a688e417b82a919f782b1ed7b, 7e54f3fcf9d24e64ab4c49314eab813a; no voters:
I20260619 16:57:41.875396 28428 raft_consensus.cc:2781] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:57:41.875540 28428 raft_consensus.cc:686] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 1 LEADER]: Becoming Leader. State: Replica: 7e54f3fcf9d24e64ab4c49314eab813a, State: Running, Role: LEADER
I20260619 16:57:41.876356 28428 consensus_queue.cc:237] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } }
I20260619 16:57:41.881784 27911 catalog_manager.cc:5495] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e54f3fcf9d24e64ab4c49314eab813a (127.16.48.1). New cstate: current_term: 1 leader_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54f3fcf9d24e64ab4c49314eab813a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 } health_report { overall_health: UNKNOWN } } }
W20260619 16:57:41.923842 28127 tablet.cc:2369] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:57:41.924223 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.918206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.16.48.1:42339)
0619 16:57:41.918425 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.918430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.918452 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:57:41.918552 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.918556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.918640 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.918803 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:57:41.919158 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.919951 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.921643 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.922072 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.922187 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:41.922853 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:57:41.923435 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:57:41.923565 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:41.923762 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 16:57:41.923846 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:57:41.933324 28358 raft_consensus.cc:1261] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [term 1 FOLLOWER]: Refusing update from remote peer 7e54f3fcf9d24e64ab4c49314eab813a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:57:41.933324 28224 raft_consensus.cc:1261] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [term 1 FOLLOWER]: Refusing update from remote peer 7e54f3fcf9d24e64ab4c49314eab813a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:57:41.934525 28430 consensus_queue.cc:1035] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [LEADER]: Connected to new peer: Peer: permanent_uuid: "07de2c6a688e417b82a919f782b1ed7b" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:57:41.934894 28428 consensus_queue.cc:1035] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8778b6c94b22457286f3eb6341261280" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:57:41.936818 28404 tablet.cc:2369] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:57:41.950691 28437 mvcc.cc:205] Tried to move back new op lower bound from 7298614320867512320 to 7298614320649748480. Current Snapshot: MvccSnapshot[applied={T|T < 7298614320867512320}]
I20260619 16:57:41.965941 27912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:57:41.966451 27912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:57:41.970970 27912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6950699d8e543cc95d7d8a1e7a7dde1: Sending DeleteTablet for 3 replicas of tablet 6054766a06394fb4bbb2c0c8411b476c
I20260619 16:57:41.972312 28338 tablet_service.cc:1514] Processing DeleteTablet for tablet 6054766a06394fb4bbb2c0c8411b476c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:57:41 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:37314
I20260619 16:57:41.972312 28204 tablet_service.cc:1514] Processing DeleteTablet for tablet 6054766a06394fb4bbb2c0c8411b476c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:57:41 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:49346
I20260619 16:57:41.973192 28061 tablet_service.cc:1514] Processing DeleteTablet for tablet 6054766a06394fb4bbb2c0c8411b476c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:57:41 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:43398
I20260619 16:57:41.973413 28446 tablet_replica.cc:330] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: stopping tablet replica
I20260619 16:57:41.973898 28446 raft_consensus.cc:2229] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:57:41.974500 28446 raft_consensus.cc:2258] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:57:41.975117 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 27993
I20260619 16:57:41.975668 28448 tablet_replica.cc:330] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: stopping tablet replica
I20260619 16:57:41.976181 28448 raft_consensus.cc:2229] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:57:41.977681 28447 tablet_replica.cc:330] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: stopping tablet replica
I20260619 16:57:41.978148 28447 raft_consensus.cc:2229] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:57:41.978777 28447 raft_consensus.cc:2258] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:57:41.977020 28448 raft_consensus.cc:2258] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:57:41.981168 28446 ts_tablet_manager.cc:1905] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:57:41.981943 28448 ts_tablet_manager.cc:1905] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:57:41.987120 28447 ts_tablet_manager.cc:1905] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:57:41.990489 28446 ts_tablet_manager.cc:1918] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:57:41.990772 28446 log.cc:1196] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/6054766a06394fb4bbb2c0c8411b476c
I20260619 16:57:41.991561 28446 ts_tablet_manager.cc:1939] T 6054766a06394fb4bbb2c0c8411b476c P 07de2c6a688e417b82a919f782b1ed7b: Deleting consensus metadata
I20260619 16:57:41.993992 27897 catalog_manager.cc:4841] TS 07de2c6a688e417b82a919f782b1ed7b (127.16.48.2:42269): tablet 6054766a06394fb4bbb2c0c8411b476c (table test-table [id=a341fdf0f70e4903b51df782d7025937]) successfully deleted
I20260619 16:57:41.994670 28447 ts_tablet_manager.cc:1918] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:57:41.994876 28447 log.cc:1196] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/6054766a06394fb4bbb2c0c8411b476c
I20260619 16:57:41.995481 28447 ts_tablet_manager.cc:1939] T 6054766a06394fb4bbb2c0c8411b476c P 8778b6c94b22457286f3eb6341261280: Deleting consensus metadata
I20260619 16:57:41.997608 27899 catalog_manager.cc:4841] TS 8778b6c94b22457286f3eb6341261280 (127.16.48.3:37273): tablet 6054766a06394fb4bbb2c0c8411b476c (table test-table [id=a341fdf0f70e4903b51df782d7025937]) successfully deleted
Jun 19 16:57:41 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:57:42.000442 28448 ts_tablet_manager.cc:1918] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:57:42.000664 28448 log.cc:1196] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/6054766a06394fb4bbb2c0c8411b476c
I20260619 16:57:42.001466 28448 ts_tablet_manager.cc:1939] T 6054766a06394fb4bbb2c0c8411b476c P 7e54f3fcf9d24e64ab4c49314eab813a: Deleting consensus metadata
I20260619 16:57:42.003767 27896 catalog_manager.cc:4841] TS 7e54f3fcf9d24e64ab4c49314eab813a (127.16.48.1:42339): tablet 6054766a06394fb4bbb2c0c8411b476c (table test-table [id=a341fdf0f70e4903b51df782d7025937]) successfully deleted
I20260619 16:57:42.008388 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:41.976493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.16.48.1:42339)
0619 16:57:41.976758 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:41.976767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:41.987308 (+ 10541us) server_negotiation.cc:407] Connection header received
0619 16:57:41.987833 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:41.987840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:41.987928 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:41.988002 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:57:41.989929 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.991140 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.993261 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:41.993935 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:41.994074 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:42.000439 (+ 6365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:57:42.000459 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:57:42.000483 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:57:42.000549 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:57:42.005010 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:42.005907 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:42.005915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:42.005917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:42.006072 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:42.006531 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:42.006535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:42.006538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:42.006917 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:57:42.007238 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:42.007585 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 16:57:42.007836 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 16:57:42.009182 28121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:57:42.117543 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 27993
I20260619 16:57:42.136495 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 28132
Jun 19 16:57:42 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:57:42.153093 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:42.137410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.16.48.2:42269)
0619 16:57:42.137640 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:42.137647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:42.137669 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:57:42.137785 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:42.137790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:42.137864 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:42.137978 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:57:42.139322 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:42.140089 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:42.141520 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:42.141945 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:42.142092 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:42.146843 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:57:42.146860 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:57:42.146877 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:57:42.146933 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:57:42.150003 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:42.150786 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:42.150794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:42.150796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:42.150940 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:42.151366 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:42.151371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:42.151374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:42.151780 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:57:42.152084 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:42.152461 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 16:57:42.152718 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:57:42.153900 28264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:57:42.246521 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 28132
I20260619 16:57:42.263447 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 28275
Jun 19 16:57:42 dist-test-slave-3ffd krb5kdc[27435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:57:42.281646 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:57:42.264387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.16.48.3:37273)
0619 16:57:42.264607 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:57:42.264612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:57:42.264668 (+ 56us) server_negotiation.cc:407] Connection header received
0619 16:57:42.264871 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:57:42.264875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:57:42.264952 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:57:42.265032 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:57:42.266731 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:42.267515 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:42.269515 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:57:42.269898 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:57:42.270005 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:57:42.275457 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:57:42.275474 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:57:42.275491 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:57:42.275553 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:57:42.278568 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:42.279427 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:42.279435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:42.279438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:42.279596 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:57:42.279987 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:57:42.279990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:57:42.279992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:57:42.280452 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:57:42.280688 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 16:57:42.281101 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 16:57:42.281278 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 16:57:42.282371 28398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:57:42.374466 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 28275
I20260619 16:57:42.392877 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 27880
I20260619 16:57:42.394065 27942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:57:42.507030 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 27880
I20260619 16:57:42.532424 16576 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:57:42.878872 16576 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:57:42.879185 16576 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:57:43.226246 16576 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:57:43.226 UTC [27465] LOG: received smart shutdown request
2026-06-19 16:57:43.230 UTC [27465] LOG: background worker "logical replication launcher" (PID 27474) exited with exit code 1
2026-06-19 16:57:43.230 UTC [27469] LOG: shutting down
2026-06-19 16:57:43.428 UTC [27465] LOG: database system is shut down
2026-06-19T16:57:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52966 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:57:43 dist-test-slave-3ffd krb5kdc[28462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:57:43 dist-test-slave-3ffd krb5kdc[28462](info): set up 2 sockets
Jun 19 16:57:43 dist-test-slave-3ffd krb5kdc[28462](info): commencing operation
krb5kdc: starting...
W20260619 16:57:44.676957 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:57:44 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:57:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/rangeradmin_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/rangeradmin_127.16.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/rangerlookup_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/rangerlookup_127.16.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/HTTP_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/HTTP_127.16.48.20@KRBTEST.COM.keytab.
I20260619 16:57:45.031358 16576 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbQ4sJq/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/postgres -l logfile start
2026-06-19 16:57:47.745 UTC [28492] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:57:47.745 UTC [28492] LOG: listening on IPv4 address "127.16.48.20", port 47873
2026-06-19 16:57:47.749 UTC [28492] LOG: listening on Unix socket "/tmp/.s.PGSQL.47873"
2026-06-19 16:57:47.770 UTC [28495] LOG: database system was shut down at 2026-06-19 16:57:46 UTC
2026-06-19 16:57:47.775 UTC [28492] LOG: database system is ready to accept connections
I20260619 16:57:48.764617 16576 mini_postgres.cc:96] Postgres bound to 47873
2026-06-19 16:57:48.782 UTC [28503] FATAL: database "slave" does not exist
127.16.48.20:47873 - accepting connections
I20260619 16:57:48.783269 16576 mini_ranger.cc:161] Starting Ranger...
I20260619 16:57:48.811394 16576 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:57:49.341111 16576 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:57:49.341221 16576 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-admin
2026-06-19 16:57:49,619 [I] DB FLAVOR :POSTGRES
2026-06-19 16:57:49,620 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:57:49,620 [I] Checking connection..
2026-06-19 16:57:49,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:57:49,872 [I] Checking connection passed.
2026-06-19 16:57:49,872 [I] --------- Verifying version history table ---------
2026-06-19 16:57:49,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:57:50,118 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:57:50,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:57:50,364 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:57:50,365 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:57:50,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:57:50.636 UTC [28567] WARNING: there is no transaction in progress
2026-06-19 16:57:50,643 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:57:50,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:57:50,902 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:57:50,902 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:57:50,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:57:51,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:57:51,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3ffd.c.gcp-upstream.internal','N') ;"
2026-06-19 16:57:51,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:57:51,930 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:57:51,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:57:52,178 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:57:52,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:57:52,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:57:52,689 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:57:52,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:57:53.778 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.797 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.815 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.833 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.852 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.872 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.933 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.941 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.959 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.971 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:53.984 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:54.004 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:54.022 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:54.031 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:54.041 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:54.050 UTC [28675] WARNING: there is no transaction in progress
2026-06-19 16:57:54,840 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:57:54,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:57:55,111 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:57:55,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:57:55,357 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:57:55,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:57:55,615 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:57:55,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:57:55,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:57:56,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3ffd.c.gcp-upstream.internal';"
2026-06-19 16:57:56,396 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:57:56,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:57:56,651 [I] DB_PATCHES have already been applied
I20260619 16:57:56.657605 16576 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbQ4sJq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:57:56.657716 16576 mini_ranger.cc:193] Using host: 127.16.48.20
I20260619 16:57:56.673669 16576 mini_ranger.cc:227] Ranger admin URL: http://127.16.48.20:50167
Jun 19, 2026 4:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:57:57.842329 16576 mini_ranger.cc:160] Time spent starting Ranger: real 9.059s user 0.000s sys 0.044s
Jun 19, 2026 4:58:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:58:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 4:58:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:58:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:58:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:58:21.179757 16576 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/rangerkms_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/rangerkms_127.16.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.48.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.48.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.48.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/HTTP_127.16.48.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.48.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/HTTP_127.16.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:58:21 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888301, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:58:21.368257 16576 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:58:21.390573 16576 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:58:22.311209 16576 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:58:22.311338 16576 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-kms
I20260619 16:58:22.311383 16576 mini_ranger_kms.cc:227] Using postgres at 127.16.48.20:47873
2026-06-19 16:58:22,681 [I] DB FLAVOR :POSTGRES
2026-06-19 16:58:22,682 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:58:22,682 [I] Checking connection
2026-06-19 16:58:22,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:58:22,951 [I] connection success
2026-06-19 16:58:22,951 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:58:22,951 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:58:22,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:58:23,218 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:58:23,218 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:58:23,218 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:58:23,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.48.20:47873/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:58:23,521 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:58:24.047199 16576 mini_ranger_kms.cc:310] Created kms service
I20260619 16:58:24.252307 16576 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:58:24.303699 16576 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:58:24 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888301, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:24.772193 16576 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:58:24.772523 16576 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbQ4sJq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbQ4sJq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:58:24.772619 16576 mini_ranger_kms.cc:240] Using host: 127.16.48.20
I20260619 16:58:24.788220 16576 mini_ranger_kms.cc:276] Ranger KMS PID: 28856
I20260619 16:58:24.788363 16576 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.48.20:60509
Jun 19, 2026 4:58:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:58:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:58:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:58:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:58:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:58:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:58:27.104667 16576 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.736s user 0.084s sys 0.307s
I20260619 16:58:27.104929 16576 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:58:27.105110 16576 mini_ranger_kms.cc:401] 127.16.48.20:60509/kms/v1/keys
Jun 19 16:58:27 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781888307, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.48.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:58:27 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888307, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.48.20@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
Jun 19 16:58:28 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888307, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.48.20@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
Jun 19 16:58:30 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:58:30.440739 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36907
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:36907
--ranger_config_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:60509/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:30.805903 28906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:30.806212 28906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:30.806368 28906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:30.814117 28906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:58:30.814261 28906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:30.814344 28906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:30.814455 28906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:58:30.814527 28906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:58:30.825441 28906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.48.20:60509/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:36907
--ranger_config_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:30.826902 28906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:30.828510 28906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:30.842628 28911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:30.842741 28912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:30.843727 28914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:30.843955 28906 server_base.cc:1034] running on GCE node
I20260619 16:58:30.844885 28906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:30.846624 28906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:30.847883 28906 hybrid_clock.cc:648] HybridClock initialized: now 1781888310847837 us; error 46 us; skew 500 ppm
I20260619 16:58:30.848394 28906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:30 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888310, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:30.856266 28906 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:58:30.858246 28906 webserver.cc:460] Webserver started at http://127.16.48.62:42405/ using document root <none> and password file <none>
I20260619 16:58:30.859030 28906 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:30.859174 28906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:30.859544 28906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:58:30 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888310, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:30.918200 28906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "9a488c0940244d6db9a2d69a813381d7"
format_stamp: "Formatted at 2026-06-19 16:58:30 on dist-test-slave-3ffd"
server_key: "5eaf0458a6b5327c27d9c888bf5200c8"
server_key_iv: "6f38d281cf8df2cb032bba95026a77fc"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:30.919198 28906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "9a488c0940244d6db9a2d69a813381d7"
format_stamp: "Formatted at 2026-06-19 16:58:30 on dist-test-slave-3ffd"
server_key: "5eaf0458a6b5327c27d9c888bf5200c8"
server_key_iv: "6f38d281cf8df2cb032bba95026a77fc"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:30.925480 28906 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 16:58:30.944365 28921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:30.946198 28906 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:58:30.946426 28906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "9a488c0940244d6db9a2d69a813381d7"
format_stamp: "Formatted at 2026-06-19 16:58:30 on dist-test-slave-3ffd"
server_key: "5eaf0458a6b5327c27d9c888bf5200c8"
server_key_iv: "6f38d281cf8df2cb032bba95026a77fc"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:30.946686 28906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:30.976081 28906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:30.981117 28906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:30.981478 28906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:31.003319 28906 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:36907
I20260619 16:58:31.003329 28972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:36907 every 8 connection(s)
I20260619 16:58:31.005017 28906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:58:31.009581 28973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:31.009892 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 28906
I20260619 16:58:31.010113 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:58:31.010990 16576 external_mini_cluster.cc:1428] Setting key 74852e728c9f18560df3e2a295782ae2
I20260619 16:58:31.031311 28973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Bootstrap starting.
I20260619 16:58:31.035808 28973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:31.037268 28973 log.cc:824] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:31.041329 28973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: No bootstrap required, opened a new log
Jun 19 16:58:31 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:31.052615 28973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } }
I20260619 16:58:31.053206 28973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:31.053356 28973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a488c0940244d6db9a2d69a813381d7, State: Initialized, Role: FOLLOWER
I20260619 16:58:31.054126 28973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } }
I20260619 16:58:31.054409 28973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:58:31.054589 28973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:58:31.054771 28973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:31.055398 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:31.013358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.16.48.62:36907)
0619 16:58:31.028444 (+ 15086us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:31.028465 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:31.028530 (+ 65us) server_negotiation.cc:407] Connection header received
0619 16:58:31.031196 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:31.031234 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:31.031863 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:31.032455 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:31.034332 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:31.036472 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:31.038467 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:31.039442 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:31.039643 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:31.045222 (+ 5579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:31.045235 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:31.045263 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:31.045323 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:31.049704 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:31.050674 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:31.050684 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:31.050694 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:31.050916 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:31.051489 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:31.051495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:31.051499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:31.052343 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:31.052641 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:31.053290 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 16:58:31.053890 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14756,"thread_start_us":10855,"threads_started":1}
I20260619 16:58:31.058223 28973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } }
I20260619 16:58:31.058827 28973 leader_election.cc:302] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a488c0940244d6db9a2d69a813381d7; no voters:
I20260619 16:58:31.059659 28973 leader_election.cc:288] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:58:31.059775 28978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:58:31.062400 28973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:58:31.064486 28978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 1 LEADER]: Becoming Leader. State: Replica: 9a488c0940244d6db9a2d69a813381d7, State: Running, Role: LEADER
I20260619 16:58:31.065469 28978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } }
I20260619 16:58:31.076692 28979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a488c0940244d6db9a2d69a813381d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } } }
I20260619 16:58:31.077131 28979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:31.077693 28980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a488c0940244d6db9a2d69a813381d7. Latest consensus state: current_term: 1 leader_uuid: "9a488c0940244d6db9a2d69a813381d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } } }
I20260619 16:58:31.077962 28980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:31.078172 28989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:58:31.087664 28989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:58:31.109768 28973 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:58:31.112736 28989 catalog_manager.cc:1347] Generated new cluster ID: 1638333f36cf448486dd514b94863e74
I20260619 16:58:31.112860 28989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:58:31.146214 28989 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:58:31.148504 28989 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:58:31.161163 28989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Generated new TSK 0
I20260619 16:58:31.162631 28989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:58:31.964461 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:60509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36907
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:58:32 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781888312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:58:32.450664 29019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:32.451223 29019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:32.451438 29019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:32.463877 29019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:32.464164 29019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:32.464387 29019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:58:32.481134 29019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.48.20:60509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:32.483027 29019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:32.484903 29019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:32.512813 29029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:32.514312 29030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:32.521966 29033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:32.522603 29019 server_base.cc:1034] running on GCE node
I20260619 16:58:32.523650 29019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:32.525905 29019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:32.529428 29019 hybrid_clock.cc:648] HybridClock initialized: now 1781888312529687 us; error 424 us; skew 500 ppm
I20260619 16:58:32.529968 29019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:32 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:32.533645 29019 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:58:32.535648 29019 webserver.cc:460] Webserver started at http://127.16.48.1:41635/ using document root <none> and password file <none>
I20260619 16:58:32.536520 29019 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:32.536657 29019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:32.536991 29019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:58:32 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:32.555754 29019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "a3ca5f597aff4e27ac87f963997a8bb8"
format_stamp: "Formatted at 2026-06-19 16:58:32 on dist-test-slave-3ffd"
server_key: "053739ef3b13bf12e4f05505fcd4abac"
server_key_iv: "95d637b899247eea888c36d0a29b9eed"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:32.557034 29019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "a3ca5f597aff4e27ac87f963997a8bb8"
format_stamp: "Formatted at 2026-06-19 16:58:32 on dist-test-slave-3ffd"
server_key: "053739ef3b13bf12e4f05505fcd4abac"
server_key_iv: "95d637b899247eea888c36d0a29b9eed"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:32.564280 29019 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 16:58:32.596243 29039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:32.598376 29019 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.008s sys 0.002s
I20260619 16:58:32.598641 29019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "a3ca5f597aff4e27ac87f963997a8bb8"
format_stamp: "Formatted at 2026-06-19 16:58:32 on dist-test-slave-3ffd"
server_key: "053739ef3b13bf12e4f05505fcd4abac"
server_key_iv: "95d637b899247eea888c36d0a29b9eed"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:32.598928 29019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:32.625528 29019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:32.631356 29019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:32.631718 29019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:32.633018 29019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:32.634963 29019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:32.635075 29019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:32.635205 29019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:32.635270 29019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:58:32 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:32.831077 29019 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:39549
I20260619 16:58:32.831120 29151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:39549 every 8 connection(s)
I20260619 16:58:32.833061 29019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:58:32.840502 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 29019
I20260619 16:58:32.840740 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:58:32.841588 16576 external_mini_cluster.cc:1428] Setting key 2f1d13c511399538ceda7f2fd6fe8186
Jun 19 16:58:32 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:32.862460 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:32.836502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:58117 (local address 127.16.48.62:36907)
0619 16:58:32.837038 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:32.837045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:32.838954 (+ 1909us) server_negotiation.cc:407] Connection header received
0619 16:58:32.841836 (+ 2882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:32.841843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:32.841953 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:32.842037 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:32.844450 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:32.845644 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:32.848534 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:32.849208 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:32.849320 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:32.854392 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:32.854400 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:32.854405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:32.854463 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:32.858132 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:32.859422 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:32.859429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:32.859433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:32.859614 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:32.860204 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:32.860210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:32.860214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:32.860559 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:32.860706 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:32.861800 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0619 16:58:32.862061 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":1647,"threads_started":1}
I20260619 16:58:32.863682 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:32.837784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36907 (local address 127.16.48.1:58117)
0619 16:58:32.838760 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:32.838813 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:32.841419 (+ 2606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:32.842659 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:32.842673 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:32.843332 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:32.844226 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:32.844261 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:32.845884 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:32.845890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:32.848290 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:32.848310 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:32.849746 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:32.849754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:32.850082 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:32.851010 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:32.851043 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:32.854057 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:32.858350 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:32.858361 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:32.858382 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:32.859259 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:32.859775 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:32.859779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:32.859783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:32.860043 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:32.860804 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:32.860813 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:32.861281 (+ 468us) client_negotiation.cc:769] Sending connection context
0619 16:58:32.861625 (+ 344us) client_negotiation.cc:240] Negotiation successful
0619 16:58:32.862210 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":400,"threads_started":1}
I20260619 16:58:32.865767 29152 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36907
I20260619 16:58:32.866153 29152 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:32.867100 29152 heartbeater.cc:511] Master 127.16.48.62:36907 requested a full tablet report, sending...
I20260619 16:58:32.870199 28938 ts_manager.cc:194] Registered new tserver with Master: a3ca5f597aff4e27ac87f963997a8bb8 (127.16.48.1:39549)
I20260619 16:58:32.872591 28938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:58117
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:58:32.920527 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:60509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36907
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:58:33 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
W20260619 16:58:33.398145 29158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:33.398525 29158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:33.398697 29158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:33.406658 29158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:33.406840 29158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:33.407016 29158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:58:33.417541 29158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.48.20:60509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:33.419076 29158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:33.420639 29158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:33.435292 29172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:33.435339 29175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:33.435289 29173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:33.436616 29158 server_base.cc:1034] running on GCE node
I20260619 16:58:33.437330 29158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:33.438419 29158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:33.439677 29158 hybrid_clock.cc:648] HybridClock initialized: now 1781888313439614 us; error 62 us; skew 500 ppm
I20260619 16:58:33.440026 29158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:33 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888313, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:33.443464 29158 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:58:33.445319 29158 webserver.cc:460] Webserver started at http://127.16.48.2:36115/ using document root <none> and password file <none>
I20260619 16:58:33.446151 29158 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:33.446309 29158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:33.446662 29158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:58:33 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888313, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:33.464017 29158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "67fd01c279bf45fe82a7f088e53478a7"
format_stamp: "Formatted at 2026-06-19 16:58:33 on dist-test-slave-3ffd"
server_key: "ce42073042c478e086922cdddc42eabd"
server_key_iv: "3e06b5c997f2c452e729a8eef5ec6780"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:33.465016 29158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "67fd01c279bf45fe82a7f088e53478a7"
format_stamp: "Formatted at 2026-06-19 16:58:33 on dist-test-slave-3ffd"
server_key: "ce42073042c478e086922cdddc42eabd"
server_key_iv: "3e06b5c997f2c452e729a8eef5ec6780"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:33.471355 29158 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:58:33.489698 29182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:33.491487 29158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:58:33.491710 29158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "67fd01c279bf45fe82a7f088e53478a7"
format_stamp: "Formatted at 2026-06-19 16:58:33 on dist-test-slave-3ffd"
server_key: "ce42073042c478e086922cdddc42eabd"
server_key_iv: "3e06b5c997f2c452e729a8eef5ec6780"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:33.492002 29158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:33.527402 29158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:33.532096 29158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:33.532465 29158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:33.533560 29158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:33.535142 29158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:33.535261 29158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:33.535425 29158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:33.535501 29158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:33.581278 29158 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:43285
I20260619 16:58:33.581297 29294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:43285 every 8 connection(s)
I20260619 16:58:33.582707 29158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:58:33.588546 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 29158
I20260619 16:58:33.588748 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:58:33.589224 16576 external_mini_cluster.cc:1428] Setting key e4682d1a68ee52caacb806f7f668c097
Jun 19 16:58:33 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888313, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:33.608451 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:33.585720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:43609 (local address 127.16.48.62:36907)
0619 16:58:33.586250 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:33.586257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:33.587542 (+ 1285us) server_negotiation.cc:407] Connection header received
0619 16:58:33.590150 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:33.590157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:33.590291 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:33.590435 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:33.592548 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:33.593730 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:33.596721 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:33.597400 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:33.597549 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:33.601724 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:33.601729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:33.601733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:33.601771 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:33.604855 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:33.605888 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:33.605893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:33.605895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:33.606035 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:33.606594 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:33.606598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:33.606600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:33.606887 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:33.607055 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:33.607786 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 16:58:33.608031 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":267,"threads_started":1}
I20260619 16:58:33.609702 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:33.586383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36907 (local address 127.16.48.2:43609)
0619 16:58:33.587322 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:33.587375 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:33.589806 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:33.590772 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:33.590783 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:33.591341 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:33.592301 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:33.592335 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:33.593923 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:33.593932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:33.596439 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:33.596477 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:33.597677 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:33.597688 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:33.598053 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:33.598960 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:33.598997 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:33.601408 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:33.605065 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:33.605077 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:33.605098 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:33.605726 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:33.606154 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:33.606160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:33.606163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:33.606455 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:33.607021 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:33.607030 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:33.607401 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 16:58:33.607679 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 16:58:33.608267 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":382,"threads_started":1}
I20260619 16:58:33.611335 29295 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36907
I20260619 16:58:33.611680 29295 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:33.612520 29295 heartbeater.cc:511] Master 127.16.48.62:36907 requested a full tablet report, sending...
I20260619 16:58:33.614046 28938 ts_manager.cc:194] Registered new tserver with Master: 67fd01c279bf45fe82a7f088e53478a7 (127.16.48.2:43285)
I20260619 16:58:33.614892 28938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:43609
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:58:33.652735 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:60509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36907
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:58:33.877689 29152 heartbeater.cc:503] Master 127.16.48.62:36907 was elected leader, sending a full tablet report...
W20260619 16:58:34.020993 29301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:34.021291 29301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:34.021464 29301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:34.029111 29301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:34.029237 29301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:34.029446 29301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:58:34.039593 29301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.48.20:60509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:34.041100 29301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:34.042469 29301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:34.056990 29306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:34.057031 29307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:34.057034 29309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:34.058032 29301 server_base.cc:1034] running on GCE node
I20260619 16:58:34.058769 29301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:34.059966 29301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:34.061219 29301 hybrid_clock.cc:648] HybridClock initialized: now 1781888314061185 us; error 76 us; skew 500 ppm
I20260619 16:58:34.061599 29301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:34 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888314, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:34.065048 29301 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:58:34.066619 29301 webserver.cc:460] Webserver started at http://127.16.48.3:40115/ using document root <none> and password file <none>
I20260619 16:58:34.067314 29301 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:34.067420 29301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:34.067698 29301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:58:34 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888314, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:34.082823 29301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "e13a4f9397e746e09b5728f13504a494"
format_stamp: "Formatted at 2026-06-19 16:58:34 on dist-test-slave-3ffd"
server_key: "7274a73b20f6903c2e46328b2ca8252b"
server_key_iv: "38ac9181724cf965708844ed3fcfc4e6"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:34.083755 29301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "e13a4f9397e746e09b5728f13504a494"
format_stamp: "Formatted at 2026-06-19 16:58:34 on dist-test-slave-3ffd"
server_key: "7274a73b20f6903c2e46328b2ca8252b"
server_key_iv: "38ac9181724cf965708844ed3fcfc4e6"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:34.089998 29301 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 16:58:34.109135 29316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:34.111181 29301 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260619 16:58:34.111377 29301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "e13a4f9397e746e09b5728f13504a494"
format_stamp: "Formatted at 2026-06-19 16:58:34 on dist-test-slave-3ffd"
server_key: "7274a73b20f6903c2e46328b2ca8252b"
server_key_iv: "38ac9181724cf965708844ed3fcfc4e6"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:34.111631 29301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:34.139659 29301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:34.145210 29301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:34.145555 29301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:34.146836 29301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:34.148785 29301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:34.148891 29301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:34.149016 29301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:34.149092 29301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:34.191531 29301 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:42121
I20260619 16:58:34.191551 29428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:42121 every 8 connection(s)
I20260619 16:58:34.193210 29301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:58:34.201790 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 29301
I20260619 16:58:34.202008 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:58:34.202491 16576 external_mini_cluster.cc:1428] Setting key 585e8d110adcba16046c18a106820f01
Jun 19 16:58:34 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888314, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:34.216030 29431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:34.196292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:60703 (local address 127.16.48.62:36907)
0619 16:58:34.196961 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:34.196965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:34.198108 (+ 1143us) server_negotiation.cc:407] Connection header received
0619 16:58:34.200025 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:34.200031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:34.200198 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:34.200321 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:34.201983 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.202901 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.205141 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.205672 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.205805 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:34.209896 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:34.209902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:34.209905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:34.209948 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:34.212843 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:34.213750 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:34.213755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:34.213758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:34.213911 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:34.214390 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:34.214394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:34.214396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:34.214701 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:34.214822 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:34.215542 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 16:58:34.215753 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":327,"threads_started":1}
I20260619 16:58:34.217852 29432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:34.197067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36907 (local address 127.16.48.3:60703)
0619 16:58:34.197913 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:34.197962 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:34.199688 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:34.200608 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:34.200618 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:34.201070 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:34.201764 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:34.201785 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.203087 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.203091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:34.204993 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:34.205008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.206304 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.206310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:34.206566 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:34.207338 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:34.207364 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:34.209586 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:34.213043 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:34.213052 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:34.213068 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:34.213591 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:34.214034 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:34.214038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:34.214040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:34.214254 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:34.214815 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:34.214822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:34.215179 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 16:58:34.215417 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 16:58:34.215843 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":384,"threads_started":1}
I20260619 16:58:34.219674 29429 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36907
I20260619 16:58:34.219970 29429 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:34.220737 29429 heartbeater.cc:511] Master 127.16.48.62:36907 requested a full tablet report, sending...
I20260619 16:58:34.222304 28937 ts_manager.cc:194] Registered new tserver with Master: e13a4f9397e746e09b5728f13504a494 (127.16.48.3:42121)
I20260619 16:58:34.223201 28937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:60703
I20260619 16:58:34.231902 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:58:34.233666 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 29019
Jun 19 16:58:34 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:58:34.260043 29434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:34.234734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.16.48.1:39549)
0619 16:58:34.235432 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:34.235440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:34.235468 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:58:34.235532 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:34.235539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:34.235781 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:34.235930 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:34.237566 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.239413 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.242065 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.242869 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.243044 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:34.250585 (+ 7541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:34.250605 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:34.250622 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:34.250696 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:34.255919 (+ 5223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:34.257242 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:34.257254 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:34.257257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:34.257509 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:34.258083 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:34.258090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:34.258104 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:34.258636 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:34.258992 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:34.259352 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 16:58:34.259656 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":387,"threads_started":1}
I20260619 16:58:34.261744 29147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:34.355239 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 29019
I20260619 16:58:34.371894 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 29158
Jun 19 16:58:34 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:58:34.389907 29436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:34.373048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.16.48.2:43285)
0619 16:58:34.373646 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:34.373653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:34.373680 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:58:34.373742 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:34.373749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:34.373963 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:34.374098 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:34.375462 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.376703 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.378024 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.378668 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.378808 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:34.383441 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:34.383458 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:34.383476 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:34.383518 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:34.386693 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:34.387548 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:34.387555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:34.387558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:34.387709 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:34.388073 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:34.388076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:34.388079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:34.388726 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:34.388984 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:34.389256 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 16:58:34.389471 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":278,"threads_started":1}
I20260619 16:58:34.391352 29290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:34.475838 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 29158
I20260619 16:58:34.496261 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 29301
Jun 19 16:58:34 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:58:34.514109 29438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:34.497235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.16.48.3:42121)
0619 16:58:34.497870 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:34.497877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:34.497906 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:58:34.497966 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:34.497972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:34.498218 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:34.498402 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:34.499718 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.500864 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.502277 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:34.502869 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:34.503035 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:34.507767 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:34.507784 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:34.507800 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:34.507841 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:34.510766 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:34.511654 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:34.511663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:34.511665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:34.511811 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:34.512288 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:34.512294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:34.512297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:34.512691 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:34.513062 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:34.513421 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 16:58:34.513722 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":344,"threads_started":1}
I20260619 16:58:34.515501 29424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:34.601377 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 29301
I20260619 16:58:34.617628 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 28906
I20260619 16:58:34.618916 28968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:34.729944 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 28906
I20260619 16:58:34.751755 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36907
--webserver_interface=127.16.48.62
--webserver_port=42405
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:36907
--ranger_config_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:60509/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.48.20:60509/kms with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:35.106405 29441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:35.106810 29441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:35.106978 29441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:35.114699 29441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:58:35.114885 29441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:35.115005 29441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:35.115108 29441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:58:35.115187 29441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:58:35.126108 29441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.48.20:60509/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:36907
--ranger_config_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:35.127605 29441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:35.129091 29441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:35.142303 29449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:35.142342 29447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:35.142390 29446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:35.148910 29441 server_base.cc:1034] running on GCE node
I20260619 16:58:35.149889 29441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:35.151597 29441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:35.152864 29441 hybrid_clock.cc:648] HybridClock initialized: now 1781888315152779 us; error 73 us; skew 500 ppm
I20260619 16:58:35.153321 29441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:35 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:35.156880 29441 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:58:35.158676 29441 webserver.cc:460] Webserver started at http://127.16.48.62:42405/ using document root <none> and password file <none>
I20260619 16:58:35.159538 29441 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:35.159709 29441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:58:35 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:35.245334 29441 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:58:35.248932 29457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:35.250633 29441 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:58:35.250903 29441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "9a488c0940244d6db9a2d69a813381d7"
format_stamp: "Formatted at 2026-06-19 16:58:30 on dist-test-slave-3ffd"
server_key: "5eaf0458a6b5327c27d9c888bf5200c8"
server_key_iv: "6f38d281cf8df2cb032bba95026a77fc"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:35.252033 29441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:35.287469 29441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:35.292497 29441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:35.292857 29441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:35.314035 29441 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:36907
I20260619 16:58:35.314007 29508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:36907 every 8 connection(s)
I20260619 16:58:35.315719 29441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:58:35.318925 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 29441
I20260619 16:58:35.319811 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:60509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:39549
--local_ip_for_outbound_sockets=127.16.48.1
--tserver_master_addrs=127.16.48.62:36907
--webserver_port=41635
--webserver_interface=127.16.48.1
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.48.20:60509/kms with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:58:35.323122 29509 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:58:35.326856 29509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Bootstrap starting.
I20260619 16:58:35.356053 29509 log.cc:824] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:35.368206 29509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:58:35.368822 29509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Bootstrap complete.
I20260619 16:58:35.378381 29509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } }
I20260619 16:58:35.379354 29509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a488c0940244d6db9a2d69a813381d7, State: Initialized, Role: FOLLOWER
I20260619 16:58:35.380086 29509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } }
I20260619 16:58:35.380391 29509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:58:35.380532 29509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:58:35.380723 29509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:58:35.385120 29509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } }
I20260619 16:58:35.385718 29509 leader_election.cc:302] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a488c0940244d6db9a2d69a813381d7; no voters:
I20260619 16:58:35.386439 29509 leader_election.cc:288] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:58:35.386592 29513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:58:35.387696 29513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [term 2 LEADER]: Becoming Leader. State: Replica: 9a488c0940244d6db9a2d69a813381d7, State: Running, Role: LEADER
I20260619 16:58:35.388336 29513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } }
I20260619 16:58:35.388952 29509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:58:35.391214 29515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a488c0940244d6db9a2d69a813381d7. Latest consensus state: current_term: 2 leader_uuid: "9a488c0940244d6db9a2d69a813381d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } } }
I20260619 16:58:35.391271 29514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9a488c0940244d6db9a2d69a813381d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a488c0940244d6db9a2d69a813381d7" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36907 } } }
I20260619 16:58:35.391654 29515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:35.391717 29514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:35.397341 29519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:58:35.405109 29519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:58:35.407127 29519 catalog_manager.cc:1259] Loaded cluster ID: 1638333f36cf448486dd514b94863e74
I20260619 16:58:35.407261 29519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:58:35.412792 29519 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:58:35.414978 29519 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Loaded TSK: 0
I20260619 16:58:35.417104 29519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:58:35.420794 29509 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 16:58:35.828346 29511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:35.828855 29511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:35.829058 29511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:35.841499 29511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:35.841675 29511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:35.841877 29511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:58:35.858705 29511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.48.20:60509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:39549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=41635
--tserver_master_addrs=127.16.48.62:36907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:35.860617 29511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:35.862713 29511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:35.897735 29547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:35.905055 29546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:35.905771 29550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:35.913473 29511 server_base.cc:1034] running on GCE node
I20260619 16:58:35.914646 29511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:35.925063 29511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:35.926404 29511 hybrid_clock.cc:648] HybridClock initialized: now 1781888315926313 us; error 97 us; skew 500 ppm
I20260619 16:58:35.926944 29511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:35 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:35.935678 29511 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:58:35.963224 29511 webserver.cc:460] Webserver started at http://127.16.48.1:41635/ using document root <none> and password file <none>
I20260619 16:58:35.964483 29511 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:35.964685 29511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:58:35 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:36.016366 29511 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.001s
I20260619 16:58:36.026039 29558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:36.028506 29511 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.003s
I20260619 16:58:36.028757 29511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "a3ca5f597aff4e27ac87f963997a8bb8"
format_stamp: "Formatted at 2026-06-19 16:58:32 on dist-test-slave-3ffd"
server_key: "053739ef3b13bf12e4f05505fcd4abac"
server_key_iv: "95d637b899247eea888c36d0a29b9eed"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:36.030355 29511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:36.066332 29511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:36.072286 29511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:36.072644 29511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:36.073974 29511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:36.075883 29511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:36.075984 29511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:36.076187 29511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:36.076285 29511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:36.144548 29511 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:39549
I20260619 16:58:36.144717 29670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:39549 every 8 connection(s)
I20260619 16:58:36.146309 29511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:58:36.150108 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 29511
I20260619 16:58:36.151058 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:60509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:43285
--local_ip_for_outbound_sockets=127.16.48.2
--tserver_master_addrs=127.16.48.62:36907
--webserver_port=36115
--webserver_interface=127.16.48.2
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.48.20:60509/kms with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:58:36 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:36.191649 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:36.150351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36907 (local address 127.16.48.1:38973)
0619 16:58:36.160844 (+ 10493us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:36.160915 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:36.165955 (+ 5040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:36.169139 (+ 3184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:36.169151 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:36.169701 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:36.170654 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:36.170691 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:36.172938 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:36.172942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:36.174488 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:36.174524 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:36.176886 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:36.176897 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:36.177139 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:36.177940 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:36.177967 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:36.180416 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:36.186209 (+ 5793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:36.186218 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:36.186233 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:36.186761 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:36.187338 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:36.187342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:36.187344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:36.187585 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:36.188667 (+ 1082us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:36.188676 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:36.189067 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 16:58:36.189401 (+ 334us) client_negotiation.cc:240] Negotiation successful
0619 16:58:36.190079 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10144,"thread_start_us":9883,"threads_started":1}
I20260619 16:58:36.191649 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:36.150229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:38973 (local address 127.16.48.62:36907)
0619 16:58:36.166765 (+ 16536us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:36.166783 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:36.166827 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:58:36.167240 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:36.167258 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:36.167755 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:36.168428 (+ 673us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:36.170974 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:36.172698 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:36.175084 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:36.176335 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:36.176601 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:36.181610 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:36.181634 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:36.181663 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:36.181730 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:36.185966 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:36.186965 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:36.186976 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:36.186987 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:36.187182 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:36.187719 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:36.187725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:36.187728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:36.188484 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:36.188734 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:36.189711 (+ 977us) server_negotiation.cc:299] Negotiation successful
0619 16:58:36.190148 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16235,"thread_start_us":18551,"threads_started":1}
I20260619 16:58:36.198670 29671 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36907
I20260619 16:58:36.199112 29671 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:36.200273 29671 heartbeater.cc:511] Master 127.16.48.62:36907 requested a full tablet report, sending...
W20260619 16:58:36.203902 29671 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:36907 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:58:36.204232 29671 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:36.204564 29671 heartbeater.cc:511] Master 127.16.48.62:36907 requested a full tablet report, sending...
I20260619 16:58:36.206027 29671 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:36.206444 29671 heartbeater.cc:511] Master 127.16.48.62:36907 requested a full tablet report, sending...
W20260619 16:58:36.207916 29671 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:58:36.600021 29675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:36.600557 29675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:36.600759 29675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:36.613104 29675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:36.613325 29675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:36.613554 29675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:58:36.629812 29675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.48.20:60509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:43285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=36115
--tserver_master_addrs=127.16.48.62:36907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:36.631726 29675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:36.633697 29675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:36.658885 29691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:36.659871 29692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:36.666417 29694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:36.667464 29675 server_base.cc:1034] running on GCE node
I20260619 16:58:36.668454 29675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:36.669952 29675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:36.671314 29675 hybrid_clock.cc:648] HybridClock initialized: now 1781888316671233 us; error 75 us; skew 500 ppm
I20260619 16:58:36.671766 29675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:36 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:36.676506 29675 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:58:36.679395 29675 webserver.cc:460] Webserver started at http://127.16.48.2:36115/ using document root <none> and password file <none>
I20260619 16:58:36.680435 29675 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:36.680627 29675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:58:36 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781888316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:58:36 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:36.725068 29675 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.005s sys 0.004s
I20260619 16:58:36.740463 29702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:36.742724 29675 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.005s
I20260619 16:58:36.743062 29675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "67fd01c279bf45fe82a7f088e53478a7"
format_stamp: "Formatted at 2026-06-19 16:58:33 on dist-test-slave-3ffd"
server_key: "ce42073042c478e086922cdddc42eabd"
server_key_iv: "3e06b5c997f2c452e729a8eef5ec6780"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:36.744689 29675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:36.793797 29675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:36.800271 29675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:36.800681 29675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:36.802073 29675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:36.804049 29675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:36.804193 29675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 16:58:36.804313 29675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:36.804399 29675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:36.880072 29675 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:43285
I20260619 16:58:36.880290 29815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:43285 every 8 connection(s)
I20260619 16:58:36.882052 29675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:58:36.888576 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 29675
I20260619 16:58:36.889446 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.48.20:60509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:42121
--local_ip_for_outbound_sockets=127.16.48.3
--tserver_master_addrs=127.16.48.62:36907
--webserver_port=40115
--webserver_interface=127.16.48.3
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.48.20:60509/kms with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:58:36 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:36.916422 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:36.886421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36907 (local address 127.16.48.2:43067)
0619 16:58:36.890856 (+ 4435us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:36.890921 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:36.893631 (+ 2710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:36.894930 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:36.894944 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:36.895653 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:36.896626 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:36.896652 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:36.897839 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:36.897845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:36.899442 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:36.899462 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:36.901063 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:36.901068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:36.901351 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:36.902167 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:36.902205 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:36.905437 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:36.910342 (+ 4905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:36.910354 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:36.910376 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:36.911168 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:36.911754 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:36.911763 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:36.911770 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:36.912114 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:36.913070 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:36.913080 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:36.913613 (+ 533us) client_negotiation.cc:769] Sending connection context
0619 16:58:36.914026 (+ 413us) client_negotiation.cc:240] Negotiation successful
0619 16:58:36.914718 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4075,"thread_start_us":5559,"threads_started":1}
I20260619 16:58:36.917341 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:36.885538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:43067 (local address 127.16.48.62:36907)
0619 16:58:36.889119 (+ 3581us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:36.889126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:36.894116 (+ 4990us) server_negotiation.cc:407] Connection header received
0619 16:58:36.894201 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:36.894207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:36.894306 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:36.894414 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:36.896879 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:36.897655 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:36.899667 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:36.900496 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:36.900621 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:36.905799 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:36.905810 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:36.905817 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:36.905872 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:36.910087 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:36.911382 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:36.911389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:36.911392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:36.911581 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:36.912335 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:36.912341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:36.912345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:36.912756 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:36.912971 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:36.916497 (+ 3526us) server_negotiation.cc:299] Negotiation successful
0619 16:58:36.916833 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3424,"thread_start_us":3282,"threads_started":1}
I20260619 16:58:36.918985 29816 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36907
I20260619 16:58:36.919377 29816 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:36.920457 29816 heartbeater.cc:511] Master 127.16.48.62:36907 requested a full tablet report, sending...
I20260619 16:58:36.923502 29474 ts_manager.cc:194] Registered new tserver with Master: 67fd01c279bf45fe82a7f088e53478a7 (127.16.48.2:43285)
I20260619 16:58:36.925899 29474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:43067
Jun 19 16:58:37 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:37.208078 29671 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:37.208606 29671 heartbeater.cc:511] Master 127.16.48.62:36907 requested a full tablet report, sending...
I20260619 16:58:37.209970 29474 ts_manager.cc:194] Registered new tserver with Master: a3ca5f597aff4e27ac87f963997a8bb8 (127.16.48.1:39549)
I20260619 16:58:37.211263 29474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:38973
W20260619 16:58:37.403662 29820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:37.404196 29820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:37.404386 29820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:37.415160 29820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:37.415417 29820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:37.415655 29820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:58:37.431334 29820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.48.20:60509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:42121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=40115
--tserver_master_addrs=127.16.48.62:36907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:37.433341 29820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:37.436691 29820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:37.472675 29825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:37.475852 29829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:37.476590 29826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:37.477203 29820 server_base.cc:1034] running on GCE node
I20260619 16:58:37.478237 29820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:37.483991 29820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:37.487560 29820 hybrid_clock.cc:648] HybridClock initialized: now 1781888317487482 us; error 65 us; skew 500 ppm
I20260619 16:58:37.488082 29820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:37 dist-test-slave-3ffd krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888317, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:37.492835 29820 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:58:37.501999 29820 webserver.cc:460] Webserver started at http://127.16.48.3:40115/ using document root <none> and password file <none>
I20260619 16:58:37.503116 29820 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:37.503304 29820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:58:37 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888317, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
Jun 19 16:58:37 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for HTTP/127.16.48.20@KRBTEST.COM
I20260619 16:58:37.554057 29820 fs_manager.cc:709] Time spent opening directory manager: real 0.025s user 0.014s sys 0.009s
I20260619 16:58:37.568514 29837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:37.570726 29820 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.000s
I20260619 16:58:37.571033 29820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "e13a4f9397e746e09b5728f13504a494"
format_stamp: "Formatted at 2026-06-19 16:58:34 on dist-test-slave-3ffd"
server_key: "7274a73b20f6903c2e46328b2ca8252b"
server_key_iv: "38ac9181724cf965708844ed3fcfc4e6"
server_key_version: "kuduclusterkey@0"
I20260619 16:58:37.572618 29820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:37.626348 29820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:37.632055 29820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:37.632485 29820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:37.640168 29820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:37.648703 29820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:37.648924 29820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 16:58:37.649094 29820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:37.649204 29820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:37.869935 29820 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:42121
I20260619 16:58:37.869998 29951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:42121 every 8 connection(s)
I20260619 16:58:37.872236 29820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:58:37.875913 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 29820
Jun 19 16:58:37 dist-test-slave-3ffd krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888317, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:37.915345 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.891502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.16.48.62:36907)
0619 16:58:37.898739 (+ 7237us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:37.898743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:37.898762 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:58:37.898817 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:37.898821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:37.898903 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:37.899070 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:37.901107 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.901875 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.904247 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.904655 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.904796 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:37.908521 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:37.908526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:37.908530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:37.908574 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:37.911908 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:37.912886 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:37.912893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:37.912896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:37.913086 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:37.913585 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:37.913590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:37.913592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:37.914001 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:37.914331 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:37.914701 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 16:58:37.914943 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7120,"thread_start_us":6943,"threads_started":1}
I20260619 16:58:37.919649 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.876072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:38001 (local address 127.16.48.62:36907)
0619 16:58:37.891526 (+ 15454us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:37.891533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:37.896182 (+ 4649us) server_negotiation.cc:407] Connection header received
0619 16:58:37.899283 (+ 3101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:37.899290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:37.899383 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:37.899468 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:37.902281 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.903770 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.906420 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.907180 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.907333 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:37.911774 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:37.911782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:37.911787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:37.911871 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:37.915595 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:37.916698 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:37.916703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:37.916706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:37.917063 (+ 357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:37.917651 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:37.917656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:37.917659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:37.918031 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:37.918183 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:37.919012 (+ 829us) server_negotiation.cc:299] Negotiation successful
0619 16:58:37.919273 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15304,"thread_start_us":15138,"threads_started":1}
I20260619 16:58:37.920583 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.876390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36907 (local address 127.16.48.3:38001)
0619 16:58:37.894256 (+ 17866us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:37.894320 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:37.898823 (+ 4503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:37.900194 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:37.900207 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:37.900877 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:37.902029 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:37.902062 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.904026 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.904031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:37.906228 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:37.906256 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.907462 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.907469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:37.907784 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:37.908566 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:37.908598 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:37.911391 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:37.915789 (+ 4398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:37.915801 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:37.915823 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:37.916531 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:37.917201 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:37.917206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:37.917209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:37.917494 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:37.918183 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:37.918194 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:37.918613 (+ 419us) client_negotiation.cc:769] Sending connection context
0619 16:58:37.918847 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 16:58:37.919293 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17462,"thread_start_us":20466,"threads_started":1}
I20260619 16:58:37.922151 29953 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36907
I20260619 16:58:37.922571 29953 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:37.923594 29953 heartbeater.cc:511] Master 127.16.48.62:36907 requested a full tablet report, sending...
I20260619 16:58:37.925591 29474 ts_manager.cc:194] Registered new tserver with Master: e13a4f9397e746e09b5728f13504a494 (127.16.48.3:42121)
I20260619 16:58:37.926470 29474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:38001
I20260619 16:58:37.930801 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:58:37.931208 29816 heartbeater.cc:503] Master 127.16.48.62:36907 was elected leader, sending a full tablet report...
I20260619 16:58:37.953666 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.937782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.16.48.62:36907)
0619 16:58:37.937986 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:37.937993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:37.938357 (+ 364us) server_negotiation.cc:407] Connection header received
0619 16:58:37.938476 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:37.938482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:37.938566 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:37.938677 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:37.940186 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.941314 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.942785 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.943448 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.943602 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:37.946723 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:37.946732 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:37.946738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:37.946789 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:37.950470 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:37.951350 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:37.951357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:37.951360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:37.951533 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:37.951993 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:37.951998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:37.952001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:37.952433 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:37.952692 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:37.953057 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 16:58:37.953322 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:58:37.959637 29474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:58:37.968076 29474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:58:37.999020 29976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.986483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:42121 (local address 127.0.0.1:35462)
0619 16:58:37.988485 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:37.988503 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:37.989643 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:37.990718 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:37.990722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:37.990753 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:37.991112 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:37.991126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.993400 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.993406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:37.995746 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:37.995762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.997881 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.997890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:37.998101 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:37.998174 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 16:58:37.998308 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:58:37.998587 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1869,"mutex_wait_us":28,"thread_start_us":1723,"threads_started":1}
I20260619 16:58:38.000507 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.987102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.16.48.3:42121)
0619 16:58:37.989106 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:37.989117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:37.989156 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:58:37.989936 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:37.989947 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:37.990217 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:37.990402 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:37.991377 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.993229 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.995941 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.997552 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.999531 (+ 1979us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:37.999835 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:37.999919 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 16:58:38.000096 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1815,"thread_start_us":1660,"threads_started":1}
I20260619 16:58:38.002393 29973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.985229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:39549 (local address 127.0.0.1:53694)
0619 16:58:37.987847 (+ 2618us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:37.987865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:37.987967 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:37.990334 (+ 2367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:37.990344 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:37.990381 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:37.990845 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:37.992286 (+ 1441us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.994325 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.994330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:37.997759 (+ 3429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:37.997774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.001469 (+ 3695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.001479 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:38.001718 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.001764 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 16:58:38.001855 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:58:38.002001 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2503,"mutex_wait_us":27,"thread_start_us":2368,"threads_started":1}
I20260619 16:58:38.003674 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.986033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.16.48.1:39549)
0619 16:58:37.989306 (+ 3273us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:37.989317 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:37.989346 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:58:37.989488 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:37.989497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:37.989806 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:37.990032 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:37.992527 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:37.994159 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:37.998017 (+ 3858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.001195 (+ 3178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.002789 (+ 1594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.003064 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:38.003140 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:58:38.003327 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3100,"thread_start_us":2939,"threads_started":1}
I20260619 16:58:38.005050 29885 tablet_service.cc:1467] Processing CreateTablet for tablet e378f7a559a84849bcb41355e6e61134 (DEFAULT_TABLE table=test-table [id=35b70b9ee83b46b794ad74944b46a287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:58:38.006747 29885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e378f7a559a84849bcb41355e6e61134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:38.008199 29606 tablet_service.cc:1467] Processing CreateTablet for tablet e378f7a559a84849bcb41355e6e61134 (DEFAULT_TABLE table=test-table [id=35b70b9ee83b46b794ad74944b46a287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:58:38.009240 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.984767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:43285 (local address 127.0.0.1:58736)
0619 16:58:37.986882 (+ 2115us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:37.986940 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:37.987181 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:38.001216 (+ 14035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:38.001222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:38.001256 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:38.001695 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:38.001708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.004276 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.004283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:38.006279 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:38.006299 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.008400 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.008407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:38.008610 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.008649 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:58:38.008723 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:58:38.008861 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1714,"mutex_wait_us":1,"thread_start_us":4379,"threads_started":1}
I20260619 16:58:38.010062 29606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e378f7a559a84849bcb41355e6e61134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:38.010960 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:37.985594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.16.48.2:43285)
0619 16:58:37.999847 (+ 14253us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:37.999857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:37.999891 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:58:37.999999 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:38.000008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:38.000337 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:38.001909 (+ 1572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:38.002160 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.004068 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.006603 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.008096 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.010044 (+ 1948us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.010357 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:38.010439 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 16:58:38.010617 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14046,"thread_start_us":14916,"threads_started":1}
I20260619 16:58:38.015587 29751 tablet_service.cc:1467] Processing CreateTablet for tablet e378f7a559a84849bcb41355e6e61134 (DEFAULT_TABLE table=test-table [id=35b70b9ee83b46b794ad74944b46a287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:58:38.017396 29751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e378f7a559a84849bcb41355e6e61134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:38.036887 29978 tablet_bootstrap.cc:492] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: Bootstrap starting.
I20260619 16:58:38.039112 29979 tablet_bootstrap.cc:492] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: Bootstrap starting.
I20260619 16:58:38.040531 29980 tablet_bootstrap.cc:492] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: Bootstrap starting.
I20260619 16:58:38.042542 29978 tablet_bootstrap.cc:654] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:38.043191 29979 tablet_bootstrap.cc:654] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:38.044018 29978 log.cc:824] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:38.044310 29980 tablet_bootstrap.cc:654] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:38.044709 29979 log.cc:824] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:38.045892 29980 log.cc:824] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:38.055764 29978 tablet_bootstrap.cc:492] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: No bootstrap required, opened a new log
I20260619 16:58:38.056216 29978 ts_tablet_manager.cc:1397] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20260619 16:58:38.056391 29980 tablet_bootstrap.cc:492] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: No bootstrap required, opened a new log
I20260619 16:58:38.056691 29980 ts_tablet_manager.cc:1397] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20260619 16:58:38.056731 29979 tablet_bootstrap.cc:492] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: No bootstrap required, opened a new log
I20260619 16:58:38.057111 29979 ts_tablet_manager.cc:1397] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20260619 16:58:38.067845 29978 raft_consensus.cc:348] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } }
I20260619 16:58:38.068475 29978 raft_consensus.cc:374] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:38.068581 29978 raft_consensus.cc:729] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ca5f597aff4e27ac87f963997a8bb8, State: Initialized, Role: FOLLOWER
I20260619 16:58:38.068015 29979 raft_consensus.cc:348] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } }
I20260619 16:58:38.068904 29979 raft_consensus.cc:374] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:38.069077 29979 raft_consensus.cc:729] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13a4f9397e746e09b5728f13504a494, State: Initialized, Role: FOLLOWER
I20260619 16:58:38.069432 29978 consensus_queue.cc:260] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } }
I20260619 16:58:38.069903 29980 raft_consensus.cc:348] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } }
I20260619 16:58:38.070492 29980 raft_consensus.cc:374] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:38.070634 29980 raft_consensus.cc:729] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67fd01c279bf45fe82a7f088e53478a7, State: Initialized, Role: FOLLOWER
I20260619 16:58:38.070348 29979 consensus_queue.cc:260] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } }
I20260619 16:58:38.071483 29980 consensus_queue.cc:260] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } }
I20260619 16:58:38.079949 29980 ts_tablet_manager.cc:1428] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: Time spent starting tablet: real 0.023s user 0.014s sys 0.004s
I20260619 16:58:38.086071 29671 heartbeater.cc:503] Master 127.16.48.62:36907 was elected leader, sending a full tablet report...
I20260619 16:58:38.086464 29978 ts_tablet_manager.cc:1428] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: Time spent starting tablet: real 0.030s user 0.014s sys 0.012s
I20260619 16:58:38.087586 29953 heartbeater.cc:503] Master 127.16.48.62:36907 was elected leader, sending a full tablet report...
I20260619 16:58:38.089960 29979 ts_tablet_manager.cc:1428] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: Time spent starting tablet: real 0.033s user 0.023s sys 0.005s
I20260619 16:58:38.117771 29986 raft_consensus.cc:482] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:58:38.118132 29986 raft_consensus.cc:504] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } }
I20260619 16:58:38.120357 29986 leader_election.cc:288] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3ca5f597aff4e27ac87f963997a8bb8 (127.16.48.1:39549), e13a4f9397e746e09b5728f13504a494 (127.16.48.3:42121)
W20260619 16:58:38.125792 29954 tablet.cc:2369] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:58:38.128691 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:38.121127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:57949 (local address 127.16.48.1:39549)
0619 16:58:38.121309 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:38.121315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:38.121827 (+ 512us) server_negotiation.cc:407] Connection header received
0619 16:58:38.122124 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:38.122130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:38.122207 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:38.122271 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:38.123115 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.124219 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.126572 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.127482 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.127578 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.127665 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:38.128258 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 16:58:38.128399 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:58:38.128749 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:38.120628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:39549 (local address 127.16.48.2:57949)
0619 16:58:38.121706 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:38.121726 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:38.121946 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:38.122478 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:38.122483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:38.122510 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:38.122914 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:38.122926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.124374 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.124384 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:38.126351 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:38.126372 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.127881 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.127885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:38.128019 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.128042 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:58:38.128227 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 16:58:38.128390 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":2582,"threads_started":1}
I20260619 16:58:38.129789 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e378f7a559a84849bcb41355e6e61134" candidate_uuid: "67fd01c279bf45fe82a7f088e53478a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" is_pre_election: true
I20260619 16:58:38.130522 29626 raft_consensus.cc:2454] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67fd01c279bf45fe82a7f088e53478a7 in term 0.
I20260619 16:58:38.130851 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:38.120909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:36017 (local address 127.16.48.3:42121)
0619 16:58:38.121115 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:38.121121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:38.123662 (+ 2541us) server_negotiation.cc:407] Connection header received
0619 16:58:38.123900 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:38.123908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:38.123976 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:38.124091 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:38.124802 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.125896 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.128491 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.129722 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.129865 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.129965 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:38.130455 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 16:58:38.130601 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:58:38.130956 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:38.120752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:42121 (local address 127.16.48.2:36017)
0619 16:58:38.123518 (+ 2766us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:38.123534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:38.123705 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:38.124104 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:38.124109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:38.124151 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:38.124600 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:38.124617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.126206 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.126212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:38.128295 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:38.128312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.129979 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.129988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:38.130182 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.130219 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:58:38.130375 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 16:58:38.130529 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2648,"thread_start_us":612,"threads_started":1}
I20260619 16:58:38.131383 29704 leader_election.cc:302] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67fd01c279bf45fe82a7f088e53478a7, a3ca5f597aff4e27ac87f963997a8bb8; no voters:
I20260619 16:58:38.131866 29986 raft_consensus.cc:2781] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:58:38.132031 29986 raft_consensus.cc:482] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:58:38.132158 29986 raft_consensus.cc:3037] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:38.132207 29905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e378f7a559a84849bcb41355e6e61134" candidate_uuid: "67fd01c279bf45fe82a7f088e53478a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13a4f9397e746e09b5728f13504a494" is_pre_election: true
I20260619 16:58:38.132762 29905 raft_consensus.cc:2454] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67fd01c279bf45fe82a7f088e53478a7 in term 0.
I20260619 16:58:38.135389 29986 raft_consensus.cc:504] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } }
I20260619 16:58:38.136353 29986 leader_election.cc:288] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [CANDIDATE]: Term 1 election: Requested vote from peers a3ca5f597aff4e27ac87f963997a8bb8 (127.16.48.1:39549), e13a4f9397e746e09b5728f13504a494 (127.16.48.3:42121)
I20260619 16:58:38.136958 29905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e378f7a559a84849bcb41355e6e61134" candidate_uuid: "67fd01c279bf45fe82a7f088e53478a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13a4f9397e746e09b5728f13504a494"
I20260619 16:58:38.136997 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e378f7a559a84849bcb41355e6e61134" candidate_uuid: "67fd01c279bf45fe82a7f088e53478a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ca5f597aff4e27ac87f963997a8bb8"
I20260619 16:58:38.137200 29905 raft_consensus.cc:3037] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:38.137256 29626 raft_consensus.cc:3037] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:38.139786 29626 raft_consensus.cc:2454] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67fd01c279bf45fe82a7f088e53478a7 in term 1.
I20260619 16:58:38.140424 29905 raft_consensus.cc:2454] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67fd01c279bf45fe82a7f088e53478a7 in term 1.
I20260619 16:58:38.140533 29704 leader_election.cc:302] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67fd01c279bf45fe82a7f088e53478a7, a3ca5f597aff4e27ac87f963997a8bb8; no voters:
W20260619 16:58:38.141194 29817 tablet.cc:2369] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:58:38.142055 29986 raft_consensus.cc:2781] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:58:38.143306 29986 raft_consensus.cc:686] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 1 LEADER]: Becoming Leader. State: Replica: 67fd01c279bf45fe82a7f088e53478a7, State: Running, Role: LEADER
I20260619 16:58:38.143883 29986 consensus_queue.cc:237] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } }
W20260619 16:58:38.149461 29672 tablet.cc:2369] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:58:38.150421 29473 catalog_manager.cc:5495] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67fd01c279bf45fe82a7f088e53478a7 (127.16.48.2). New cstate: current_term: 1 leader_uuid: "67fd01c279bf45fe82a7f088e53478a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fd01c279bf45fe82a7f088e53478a7" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260619 16:58:38.206348 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:38.199545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.16.48.2:43285)
0619 16:58:38.199765 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:38.199771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:38.199805 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:58:38.199930 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:38.199936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:38.200059 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:38.200282 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:58:38.200844 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.201875 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.203387 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.203950 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.204085 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.204900 (+ 815us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:58:38.205677 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:58:38.205832 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:38.205957 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:58:38.206052 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:58:38.216784 29626 raft_consensus.cc:1261] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [term 1 FOLLOWER]: Refusing update from remote peer 67fd01c279bf45fe82a7f088e53478a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:58:38.216779 29905 raft_consensus.cc:1261] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [term 1 FOLLOWER]: Refusing update from remote peer 67fd01c279bf45fe82a7f088e53478a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:58:38.218072 29989 consensus_queue.cc:1035] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3ca5f597aff4e27ac87f963997a8bb8" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:58:38.218463 29986 consensus_queue.cc:1035] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13a4f9397e746e09b5728f13504a494" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:58:38.236754 29992 mvcc.cc:205] Tried to move back new op lower bound from 7298614551403778048 to 7298614551122370560. Current Snapshot: MvccSnapshot[applied={T|T < 7298614551403778048}]
I20260619 16:58:38.237447 29994 mvcc.cc:205] Tried to move back new op lower bound from 7298614551403778048 to 7298614551122370560. Current Snapshot: MvccSnapshot[applied={T|T < 7298614551403778048}]
I20260619 16:58:38.242890 29993 mvcc.cc:205] Tried to move back new op lower bound from 7298614551403778048 to 7298614551122370560. Current Snapshot: MvccSnapshot[applied={T|T < 7298614551403778048}]
I20260619 16:58:38.264560 29474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:58:38.264897 29474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:58:38.269156 29474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a488c0940244d6db9a2d69a813381d7: Sending DeleteTablet for 3 replicas of tablet e378f7a559a84849bcb41355e6e61134
I20260619 16:58:38.270534 29606 tablet_service.cc:1514] Processing DeleteTablet for tablet e378f7a559a84849bcb41355e6e61134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:58:38 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:53694
I20260619 16:58:38.270566 29751 tablet_service.cc:1514] Processing DeleteTablet for tablet e378f7a559a84849bcb41355e6e61134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:58:38 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:58736
I20260619 16:58:38.271502 30001 tablet_replica.cc:330] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: stopping tablet replica
I20260619 16:58:38.272071 30001 raft_consensus.cc:2229] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:58:38.272071 29885 tablet_service.cc:1514] Processing DeleteTablet for tablet e378f7a559a84849bcb41355e6e61134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:58:38 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:35462
I20260619 16:58:38.272811 30001 raft_consensus.cc:2258] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:58:38.275159 30001 ts_tablet_manager.cc:1905] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:58:38.280136 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 29511
I20260619 16:58:38.280951 30002 tablet_replica.cc:330] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: stopping tablet replica
I20260619 16:58:38.282155 30002 raft_consensus.cc:2229] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:58:38.285920 30003 tablet_replica.cc:330] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: stopping tablet replica
I20260619 16:58:38.287024 30001 ts_tablet_manager.cc:1918] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:58:38.287235 30001 log.cc:1196] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/e378f7a559a84849bcb41355e6e61134
I20260619 16:58:38.287854 30001 ts_tablet_manager.cc:1939] T e378f7a559a84849bcb41355e6e61134 P a3ca5f597aff4e27ac87f963997a8bb8: Deleting consensus metadata
I20260619 16:58:38.286445 30003 raft_consensus.cc:2229] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:58:38.288069 30002 raft_consensus.cc:2258] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:58:38.290094 29459 catalog_manager.cc:4841] TS a3ca5f597aff4e27ac87f963997a8bb8 (127.16.48.1:39549): tablet e378f7a559a84849bcb41355e6e61134 (table test-table [id=35b70b9ee83b46b794ad74944b46a287]) successfully deleted
I20260619 16:58:38.291524 30002 ts_tablet_manager.cc:1905] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:58:38.288604 30003 raft_consensus.cc:2258] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:58:38.299067 30003 ts_tablet_manager.cc:1905] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:58:38.304008 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:38.283275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.16.48.1:39549)
0619 16:58:38.283479 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:38.283485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:38.283510 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:58:38.283582 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:38.283587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:38.283666 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:38.283793 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:38.286547 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.287790 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.289921 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.290598 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.290757 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.294849 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:38.294868 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:38.294888 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:38.294935 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:38.299139 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:38.300386 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:38.300399 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:38.300404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:38.300709 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:38.301254 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:38.301260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:38.301263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:38.301753 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:38.302071 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:38.303334 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0619 16:58:38.303608 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:58:38.304772 29666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:38.326094 30002 ts_tablet_manager.cc:1918] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:58:38.326339 30002 log.cc:1196] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/e378f7a559a84849bcb41355e6e61134
I20260619 16:58:38.326900 30002 ts_tablet_manager.cc:1939] T e378f7a559a84849bcb41355e6e61134 P 67fd01c279bf45fe82a7f088e53478a7: Deleting consensus metadata
I20260619 16:58:38.328964 29460 catalog_manager.cc:4841] TS 67fd01c279bf45fe82a7f088e53478a7 (127.16.48.2:43285): tablet e378f7a559a84849bcb41355e6e61134 (table test-table [id=35b70b9ee83b46b794ad74944b46a287]) successfully deleted
I20260619 16:58:38.340739 30003 ts_tablet_manager.cc:1918] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:58:38.340967 30003 log.cc:1196] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/e378f7a559a84849bcb41355e6e61134
I20260619 16:58:38.341555 30003 ts_tablet_manager.cc:1939] T e378f7a559a84849bcb41355e6e61134 P e13a4f9397e746e09b5728f13504a494: Deleting consensus metadata
I20260619 16:58:38.343611 29458 catalog_manager.cc:4841] TS e13a4f9397e746e09b5728f13504a494 (127.16.48.3:42121): tablet e378f7a559a84849bcb41355e6e61134 (table test-table [id=35b70b9ee83b46b794ad74944b46a287]) successfully deleted
I20260619 16:58:38.418071 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 29511
I20260619 16:58:38.435755 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 29675
I20260619 16:58:38.453425 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:38.436741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.16.48.2:43285)
0619 16:58:38.436919 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:38.436923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:38.436980 (+ 57us) server_negotiation.cc:407] Connection header received
0619 16:58:38.437196 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:38.437200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:38.437269 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:38.437407 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:38.439145 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.440000 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.442040 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.442515 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.442624 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.446312 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:38.446328 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:38.446345 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:38.446384 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:38.450132 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:38.451151 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:38.451159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:38.451161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:38.451330 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:38.451745 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:38.451748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:38.451751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:38.452295 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:38.452559 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:38.452901 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:58:38.453090 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:58:38.454218 29811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:38.572363 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 29675
I20260619 16:58:38.592765 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 29820
I20260619 16:58:38.607183 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:38.593542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.16.48.3:42121)
0619 16:58:38.593727 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:38.593732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:38.593751 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:58:38.593940 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:38.593945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:38.594028 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:38.594119 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:38.595422 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.596279 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.597553 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:38.597934 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:38.598041 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:38.601311 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:38.601329 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:38.601346 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:38.601392 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:38.604340 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:38.605134 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:38.605142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:38.605145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:38.605307 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:38.605673 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:38.605676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:38.605678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:38.606033 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:38.606281 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:38.606615 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 16:58:38.606789 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:58:38.607820 29947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:38.698165 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 29820
I20260619 16:58:38.718308 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 29441
I20260619 16:58:38.719564 29504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:38.834406 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 29441
I20260619 16:58:38.857025 16576 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:58:39.193421 16576 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:58:39.193687 16576 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:58:39.540361 16576 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:58:39.540 UTC [28492] LOG: received smart shutdown request
2026-06-19 16:58:39.543 UTC [28492] LOG: background worker "logical replication launcher" (PID 28501) exited with exit code 1
2026-06-19 16:58:39.544 UTC [28496] LOG: shutting down
2026-06-19 16:58:39.726 UTC [28492] LOG: database system is shut down
2026-06-19T16:58:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56304 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:58:39 dist-test-slave-3ffd krb5kdc[30016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:58:39 dist-test-slave-3ffd krb5kdc[30016](info): set up 2 sockets
Jun 19 16:58:39 dist-test-slave-3ffd krb5kdc[30016](info): commencing operation
krb5kdc: starting...
W20260619 16:58:40.968928 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:58:41 dist-test-slave-3ffd krb5kdc[30016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:58:41.223590 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:45313
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:45313,127.16.48.61:45571,127.16.48.60:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:41.589011 30029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:41.589531 30029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:41.589731 30029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:41.598373 30029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:58:41.598510 30029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:41.598558 30029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:41.598618 30029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:58:41.598661 30029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:58:41.609050 30029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:45313,127.16.48.61:45571,127.16.48.60:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:41.610419 30029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:41.611725 30029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:41.624958 30035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:41.624917 30037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:41.624981 30034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:41.626019 30029 server_base.cc:1034] running on GCE node
I20260619 16:58:41.626950 30029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:41.628669 30029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:41.629921 30029 hybrid_clock.cc:648] HybridClock initialized: now 1781888321629866 us; error 44 us; skew 500 ppm
I20260619 16:58:41.630399 30029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:41 dist-test-slave-3ffd krb5kdc[30016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:41.639467 30029 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:58:41.641369 30029 webserver.cc:460] Webserver started at http://127.16.48.62:39255/ using document root <none> and password file <none>
I20260619 16:58:41.642199 30029 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:41.642354 30029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:41.642712 30029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:58:41.646214 30029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "72485882fcde48d286eed71c5200d850"
format_stamp: "Formatted at 2026-06-19 16:58:41 on dist-test-slave-3ffd"
server_key: "d6b0cb1bee1edc0808cfd65a82cd90b1"
server_key_iv: "8eb99a79223f65e89665e7ef1f2d00cb"
server_key_version: "encryptionkey@0"
I20260619 16:58:41.647230 30029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "72485882fcde48d286eed71c5200d850"
format_stamp: "Formatted at 2026-06-19 16:58:41 on dist-test-slave-3ffd"
server_key: "d6b0cb1bee1edc0808cfd65a82cd90b1"
server_key_iv: "8eb99a79223f65e89665e7ef1f2d00cb"
server_key_version: "encryptionkey@0"
I20260619 16:58:41.653322 30029 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 16:58:41.657452 30044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:41.659183 30029 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:58:41.659355 30029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "72485882fcde48d286eed71c5200d850"
format_stamp: "Formatted at 2026-06-19 16:58:41 on dist-test-slave-3ffd"
server_key: "d6b0cb1bee1edc0808cfd65a82cd90b1"
server_key_iv: "8eb99a79223f65e89665e7ef1f2d00cb"
server_key_version: "encryptionkey@0"
I20260619 16:58:41.659567 30029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:41.696053 30029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:41.700889 30029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:41.701169 30029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:41.721228 30029 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:45313
I20260619 16:58:41.721261 30095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:45313 every 8 connection(s)
I20260619 16:58:41.722766 30029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:58:41.726853 30096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:41.729385 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 30029
I20260619 16:58:41.729633 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:58:41.730280 16576 external_mini_cluster.cc:1428] Setting key fc9ae131c434f62222e5fc70a8e7ba9b
I20260619 16:58:41.734341 30096 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:58:41 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.48.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.61@KRBTEST.COM" created.
I20260619 16:58:41.760493 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.739243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.16.48.62:45313)
0619 16:58:41.740023 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:41.740033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:41.740101 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:58:41.741487 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:41.741499 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:41.741858 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:41.742092 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:41.743347 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:41.745416 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:41.746938 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:41.748035 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:41.748316 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:41.751984 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:41.751994 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:41.752010 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:41.752045 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:41.755563 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:41.756693 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:41.756700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:41.756703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:41.756869 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:41.757366 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:41.757370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:41.757372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:41.757964 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:41.758132 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:41.758880 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 16:58:41.759220 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":545,"thread_start_us":340,"threads_started":1}
I20260619 16:58:41.760584 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.737788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:45313 (local address 127.0.0.1:41830)
0619 16:58:41.738693 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:41.738745 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:41.741028 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:41.742143 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:41.742148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:41.742736 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:41.743120 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:41.743140 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:41.745595 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:41.745599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:41.746754 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:41.746769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:41.748332 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:41.748337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:41.748571 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:41.749306 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:41.749331 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:41.751724 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:41.755748 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:41.755765 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:41.755783 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:41.756519 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:41.757015 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:41.757022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:41.757029 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:41.757253 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:41.758143 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:41.758155 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:41.758448 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 16:58:41.758760 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 16:58:41.759417 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":125,"thread_start_us":397,"threads_started":1}
I20260619 16:58:41.764506 30096 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:58:41.765789 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.765325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60040)
0619 16:58:41.765518 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:41.765660 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 16:58:41.766283 30048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
I20260619 16:58:41.768287 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.767711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60056)
0619 16:58:41.767904 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:41.768018 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 16:58:41.768613 30096 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:45571: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab.
Entry for principal kudu/127.16.48.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab.
I20260619 16:58:41.788789 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.61:45571
--webserver_interface=127.16.48.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab
--principal=kudu/127.16.48.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:45313,127.16.48.61:45571,127.16.48.60:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:58:41.818017 30096 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } attempt: 1
I20260619 16:58:41.819341 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.818871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60060)
0619 16:58:41.819083 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:41.819209 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 16:58:41.820611 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.820185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60066)
0619 16:58:41.820359 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:41.820436 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:58:41.820816 30096 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:45571: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
I20260619 16:58:41.889111 30096 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } attempt: 2
I20260619 16:58:41.890336 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.889876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60076)
0619 16:58:41.890088 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:41.890191 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 16:58:41.891456 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.891068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60086)
0619 16:58:41.891229 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:41.891313 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 16:58:41.891655 30096 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:45571: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
I20260619 16:58:41.982930 30096 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } attempt: 3
I20260619 16:58:41.984411 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.983789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60098)
0619 16:58:41.984025 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:41.984253 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 16:58:41.985675 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:41.985248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60100)
0619 16:58:41.985434 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:41.985510 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:58:41.985869 30096 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:45571: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
I20260619 16:58:42.129143 30096 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } attempt: 4
I20260619 16:58:42.130585 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.129966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60110)
0619 16:58:42.130323 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.130417 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260619 16:58:42.131906 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.131465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60126)
0619 16:58:42.131646 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.131756 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 16:58:42.132148 30096 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:45571: Network error: Client connection negotiation failed: client connection to 127.16.48.61:45571: connect: Connection refused (error 111)
W20260619 16:58:42.144340 30103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:42.144708 30103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:42.144874 30103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:42.152567 30103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:58:42.152742 30103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:42.152860 30103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:42.152963 30103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:58:42.153049 30103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:58:42.163710 30103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:45313,127.16.48.61:45571,127.16.48.60:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab
--principal=kudu/127.16.48.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.61:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.48.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:42.165402 30103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:42.166932 30103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:42.180081 30109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:42.180150 30108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:42.180056 30111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:42.180904 30103 server_base.cc:1034] running on GCE node
I20260619 16:58:42.181882 30103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:42.183478 30103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:42.184712 30103 hybrid_clock.cc:648] HybridClock initialized: now 1781888322184661 us; error 42 us; skew 500 ppm
I20260619 16:58:42.185108 30103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888322, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:42.188966 30103 init.cc:377] Logged in from keytab as kudu/127.16.48.61@KRBTEST.COM (short username kudu)
I20260619 16:58:42.190824 30103 webserver.cc:460] Webserver started at http://127.16.48.61:44835/ using document root <none> and password file <none>
I20260619 16:58:42.191672 30103 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:42.191815 30103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:42.192230 30103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:58:42.195673 30103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/data/instance:
uuid: "326f6c1e2a514882b2f940ecf52a7dbe"
format_stamp: "Formatted at 2026-06-19 16:58:42 on dist-test-slave-3ffd"
server_key: "21cdc72763f79f286b39d74eba3e1561"
server_key_iv: "667d52e5c5f3fe5381b0212ab8ba0774"
server_key_version: "encryptionkey@0"
I20260619 16:58:42.196713 30103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/wal/instance:
uuid: "326f6c1e2a514882b2f940ecf52a7dbe"
format_stamp: "Formatted at 2026-06-19 16:58:42 on dist-test-slave-3ffd"
server_key: "21cdc72763f79f286b39d74eba3e1561"
server_key_iv: "667d52e5c5f3fe5381b0212ab8ba0774"
server_key_version: "encryptionkey@0"
I20260619 16:58:42.203078 30103 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260619 16:58:42.207329 30118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:42.209033 30103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:58:42.209275 30103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/wal
uuid: "326f6c1e2a514882b2f940ecf52a7dbe"
format_stamp: "Formatted at 2026-06-19 16:58:42 on dist-test-slave-3ffd"
server_key: "21cdc72763f79f286b39d74eba3e1561"
server_key_iv: "667d52e5c5f3fe5381b0212ab8ba0774"
server_key_version: "encryptionkey@0"
I20260619 16:58:42.209488 30103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:42.221757 30103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:42.226578 30103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:42.226912 30103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:42.246824 30103 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.61:45571
I20260619 16:58:42.246856 30169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.61:45571 every 8 connection(s)
I20260619 16:58:42.248435 30103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
I20260619 16:58:42.252717 30170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:42.253432 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 30103
I20260619 16:58:42.253646 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-1/wal/instance
I20260619 16:58:42.254217 16576 external_mini_cluster.cc:1428] Setting key 0be7ed0d49ddb5024113fd6490143f4b
I20260619 16:58:42.263947 30170 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.48.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.60@KRBTEST.COM" created.
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888322, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:42.292079 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.267310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.16.48.62:45313)
0619 16:58:42.267856 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.267863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.269197 (+ 1334us) server_negotiation.cc:407] Connection header received
0619 16:58:42.272585 (+ 3388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.272590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.272698 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.272800 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.275333 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.276069 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.277777 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.278202 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.278349 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.282973 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.282984 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.282991 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.283061 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.287816 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.289090 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.289099 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.289106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.289307 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.289922 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.289927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.289929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.290249 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.290381 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.291582 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.291784 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":274,"threads_started":1}
I20260619 16:58:42.293857 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.267968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:45313 (local address 127.0.0.1:41832)
0619 16:58:42.268974 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.269040 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:42.272140 (+ 3100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:42.273164 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:42.273180 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:42.274117 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:42.275078 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.275109 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.276313 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.276318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.277605 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.277619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.278460 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.278467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.278711 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.279431 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:42.279462 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:42.282625 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:42.288080 (+ 5455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.288090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.288108 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.288841 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.289476 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.289480 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.289483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.289718 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.290409 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:42.290421 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:42.291058 (+ 637us) client_negotiation.cc:769] Sending connection context
0619 16:58:42.291453 (+ 395us) client_negotiation.cc:240] Negotiation successful
0619 16:58:42.292147 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":416,"threads_started":1}
I20260619 16:58:42.296029 30170 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888322, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 16:58:42.314047 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.297034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60132)
0619 16:58:42.297297 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.297325 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:42.297466 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:42.298696 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:42.298704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:42.299275 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:42.299503 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.299518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.301029 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.301038 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.302621 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.302642 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.303553 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.303561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.303769 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.304505 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:42.304535 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:42.307303 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:42.311083 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.311087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.311091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.311495 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.311888 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.311892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.311895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.312243 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.313044 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:42.313050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:42.313232 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 16:58:42.313402 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 16:58:42.313664 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 16:58:42.314404 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.297246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.16.48.61:45571)
0619 16:58:42.298169 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.298176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.298205 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:58:42.298257 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.298263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.298487 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.298628 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.299676 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.300868 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.302790 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.303377 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.303482 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.307614 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.307631 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.307649 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.307684 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.310919 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.311625 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.311633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.311635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.311755 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.312375 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.312379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.312381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.312848 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.313132 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.313645 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.313913 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":597,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab.
Entry for principal kudu/127.16.48.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab.
I20260619 16:58:42.316475 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.60:40179
--webserver_interface=127.16.48.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab
--principal=kudu/127.16.48.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:45313,127.16.48.61:45571,127.16.48.60:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:58:42.317010 30170 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:58:42.318358 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.317722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60170)
0619 16:58:42.317963 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.318145 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 16:58:42.318743 30119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
I20260619 16:58:42.320587 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.320070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60172)
0619 16:58:42.320293 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.320414 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 16:58:42.320851 30170 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:40179: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
I20260619 16:58:42.370252 30170 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } attempt: 1
I20260619 16:58:42.371533 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.371043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60182)
0619 16:58:42.371257 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.371367 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:58:42.372689 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.372313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60198)
0619 16:58:42.372473 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.372560 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:58:42.372977 30170 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:40179: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
I20260619 16:58:42.431468 30096 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } attempt: 5
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 16:58:42.441233 30170 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } attempt: 2
I20260619 16:58:42.442466 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.441901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60206)
0619 16:58:42.442161 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.442272 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 16:58:42.443684 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.443276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60216)
0619 16:58:42.443444 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.443536 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 16:58:42.444043 30170 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:40179: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
I20260619 16:58:42.448459 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.432420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.16.48.61:45571)
0619 16:58:42.432639 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.432653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.432686 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:58:42.432929 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.432934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.433027 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.433125 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.434189 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.434891 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.436751 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.437224 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.437339 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.441477 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.441487 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.441493 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.441553 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.445256 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.446169 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.446176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.446180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.446332 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.446803 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.446809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.446813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.447149 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.447323 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.447792 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.448036 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:58:42.448458 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.432359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60148)
0619 16:58:42.432605 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.432630 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:42.432766 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:42.433210 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:42.433215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:42.433735 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:42.433968 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.433981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.435060 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.435066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.436550 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.436573 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.437407 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.437413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.437623 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.438374 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:42.438399 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:42.441130 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:42.445496 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.445510 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.445514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.446003 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.446489 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.446501 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.446504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.446664 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.447283 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:42.447288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:42.447465 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 16:58:42.447659 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 16:58:42.448022 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 16:58:42.449656 30096 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:58:42.451005 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.450529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60222)
0619 16:58:42.450758 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.450858 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 16:58:42.452093 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.451742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60228)
0619 16:58:42.451904 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.451980 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 16:58:42.452370 30096 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:40179: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
I20260619 16:58:42.503631 30096 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } attempt: 1
I20260619 16:58:42.504998 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.504496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60244)
0619 16:58:42.504730 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.504825 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 16:58:42.506131 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.505781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60252)
0619 16:58:42.505948 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.506021 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 16:58:42.506353 30096 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:40179: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
I20260619 16:58:42.535355 30170 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } attempt: 3
I20260619 16:58:42.536782 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.536236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60258)
0619 16:58:42.536500 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.536601 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:58:42.538095 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.537598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60270)
0619 16:58:42.537876 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.537955 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260619 16:58:42.538355 30170 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:40179: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
I20260619 16:58:42.574642 30096 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } attempt: 2
I20260619 16:58:42.575843 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.575331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60278)
0619 16:58:42.575575 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.575673 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 16:58:42.577113 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.576733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60292)
0619 16:58:42.576899 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.576986 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:58:42.577327 30096 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:40179: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
W20260619 16:58:42.671846 30178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:42.672271 30178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:42.672461 30178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:42.680075 30178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:58:42.680308 30178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:42.680428 30178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:42.680539 30178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:58:42.680631 30178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:58:42.681602 30170 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } attempt: 4
I20260619 16:58:42.682806 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.682307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60300)
0619 16:58:42.682561 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.682647 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 16:58:42.683564 30096 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } attempt: 3
I20260619 16:58:42.684034 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.683577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60316)
0619 16:58:42.683771 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.683866 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 16:58:42.684381 30170 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:40179: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
I20260619 16:58:42.684664 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.684212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60318)
0619 16:58:42.684377 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.684497 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:58:42.685802 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.685357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60322)
0619 16:58:42.685578 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.685669 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 16:58:42.686045 30096 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:40179: Network error: Client connection negotiation failed: client connection to 127.16.48.60:40179: connect: Connection refused (error 111)
I20260619 16:58:42.692443 30178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:45313,127.16.48.61:45571,127.16.48.60:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab
--principal=kudu/127.16.48.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.60:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.48.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:42.693823 30178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:42.695165 30178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:42.710680 30186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:42.710644 30184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:42.710830 30183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:42.712178 30178 server_base.cc:1034] running on GCE node
I20260619 16:58:42.713151 30178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:42.714924 30178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:42.716264 30178 hybrid_clock.cc:648] HybridClock initialized: now 1781888322716225 us; error 38 us; skew 500 ppm
I20260619 16:58:42.716737 30178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888322, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:42.720193 30178 init.cc:377] Logged in from keytab as kudu/127.16.48.60@KRBTEST.COM (short username kudu)
I20260619 16:58:42.722051 30178 webserver.cc:460] Webserver started at http://127.16.48.60:37917/ using document root <none> and password file <none>
I20260619 16:58:42.722837 30178 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:42.722980 30178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:42.723337 30178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:58:42.726698 30178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/data/instance:
uuid: "a209c19d91a84b3abe78e3ab7983a83a"
format_stamp: "Formatted at 2026-06-19 16:58:42 on dist-test-slave-3ffd"
server_key: "021b41ec60abc4190f4a8214df0dc82e"
server_key_iv: "135f5a76b38b9c0c5bcc7523c34b310f"
server_key_version: "encryptionkey@0"
I20260619 16:58:42.727679 30178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/wal/instance:
uuid: "a209c19d91a84b3abe78e3ab7983a83a"
format_stamp: "Formatted at 2026-06-19 16:58:42 on dist-test-slave-3ffd"
server_key: "021b41ec60abc4190f4a8214df0dc82e"
server_key_iv: "135f5a76b38b9c0c5bcc7523c34b310f"
server_key_version: "encryptionkey@0"
I20260619 16:58:42.733799 30178 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.005s
I20260619 16:58:42.737821 30193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:42.739401 30178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:58:42.739629 30178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/wal
uuid: "a209c19d91a84b3abe78e3ab7983a83a"
format_stamp: "Formatted at 2026-06-19 16:58:42 on dist-test-slave-3ffd"
server_key: "021b41ec60abc4190f4a8214df0dc82e"
server_key_iv: "135f5a76b38b9c0c5bcc7523c34b310f"
server_key_version: "encryptionkey@0"
I20260619 16:58:42.739882 30178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:42.764011 30178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:42.768741 30178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:42.769054 30178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:42.788939 30178 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.60:40179
I20260619 16:58:42.788957 30244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.60:40179 every 8 connection(s)
I20260619 16:58:42.790557 30178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
I20260619 16:58:42.791221 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 30178
I20260619 16:58:42.791450 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781888107067647-16576-0/minicluster-data/master-2/wal/instance
I20260619 16:58:42.792064 16576 external_mini_cluster.cc:1428] Setting key 28316bc64a81ee332560a83ef527e204
I20260619 16:58:42.795708 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:42.805212 30245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888322, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:42.818707 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.794248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.16.48.60:40179)
0619 16:58:42.795105 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.795123 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.795178 (+ 55us) server_negotiation.cc:407] Connection header received
0619 16:58:42.797624 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.797653 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.798198 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.798728 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.800093 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.802253 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.803928 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.805030 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.805263 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.810036 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.810043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.810065 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.810101 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.813472 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.814367 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.814379 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.814387 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.814576 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.815023 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.815028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.815037 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.815949 (+ 912us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.816342 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.816819 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.817284 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":381,"threads_started":1}
I20260619 16:58:42.826043 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.808442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.16.48.62:45313)
0619 16:58:42.809125 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.809132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.809681 (+ 549us) server_negotiation.cc:407] Connection header received
0619 16:58:42.809877 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.809886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.809976 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.810107 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.811428 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.812272 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.813952 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.814463 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.814626 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.818427 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.818433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.818436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.818473 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.822943 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.823719 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.823726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.823729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.823868 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.824363 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.824368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.824370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.824672 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.824887 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.825382 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.825629 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":332,"threads_started":1}
I20260619 16:58:42.826062 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.808456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:45313 (local address 127.0.0.1:41838)
0619 16:58:42.809509 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.809563 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:42.809735 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:42.810165 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:42.810177 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:42.810912 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:42.811247 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.811257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.812455 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.812460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.813800 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.813813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.814678 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.814685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.814873 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.815592 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:42.815642 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:42.818196 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:42.823090 (+ 4894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.823117 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.823123 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.823583 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.823990 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.823995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.824001 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.824230 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.824837 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:42.824846 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:42.825039 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 16:58:42.825247 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 16:58:42.825616 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":643,"threads_started":1}
I20260619 16:58:42.827874 30245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888322, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 16:58:42.839788 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.822293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.16.48.62:45313)
0619 16:58:42.822959 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.822963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.822983 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:58:42.823037 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.823046 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.823101 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.823226 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.824691 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.825590 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.827290 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.827875 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.828010 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.833206 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.833212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.833216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.833258 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.836829 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.837610 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.837617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.837621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.837776 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.838200 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.838206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.838209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.838530 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.838836 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.839146 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.839395 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":314,"threads_started":1}
I20260619 16:58:42.843611 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.828841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:45571 (local address 127.0.0.1:60164)
0619 16:58:42.829132 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.829152 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:42.829285 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:42.829643 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:42.829649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:42.830133 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:42.830344 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.830360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.831398 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.831404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.832456 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.832470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.833418 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.833425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.833642 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.834269 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:42.834301 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:42.836809 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:42.840216 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.840222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.840226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.840831 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.841434 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.841442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.841446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.841684 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.842493 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:42.842497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:42.842743 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 16:58:42.842905 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 16:58:42.843259 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260619 16:58:42.843863 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.828815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.16.48.61:45571)
0619 16:58:42.828989 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.828994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.829229 (+ 235us) server_negotiation.cc:407] Connection header received
0619 16:58:42.829398 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.829402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.829465 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.829548 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.830535 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.831241 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.832623 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.833247 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.833346 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.837061 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.837067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.837070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.837108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.840036 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.840977 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.840983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.840986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.841208 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.841856 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.841862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.841865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.842194 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.842369 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.843053 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.843378 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:58:42.844933 30245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888322, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
I20260619 16:58:42.859689 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.845797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.16.48.60:40179)
0619 16:58:42.845987 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.845992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.846185 (+ 193us) server_negotiation.cc:407] Connection header received
0619 16:58:42.846355 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.846360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.846438 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.846541 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.847232 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.848182 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.849768 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.850457 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.850601 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.853950 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.853954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.853957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.853988 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.856898 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.857594 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.857602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.857605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.857748 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.858100 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.858107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.858111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.858377 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.858532 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.858968 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.859263 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":64}
I20260619 16:58:42.859843 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.841983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.16.48.61:45571)
0619 16:58:42.842713 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.842719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.842741 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:58:42.842789 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.842793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.842905 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.843007 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.845177 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.846209 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.847850 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.848484 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.848642 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.853933 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.853939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.853943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.853982 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.856925 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.857737 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.857745 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.857748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.857891 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.858299 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.858304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.858309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.858584 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.858865 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.859175 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.859417 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":360,"threads_started":1}
I20260619 16:58:42.860011 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.845797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60348)
0619 16:58:42.846089 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.846106 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:42.846231 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:42.846579 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:42.846582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:42.846939 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:42.847090 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.847098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.848326 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.848332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.849612 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.849625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.850661 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.850665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.850822 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.851426 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:42.851443 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:42.853717 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:42.857056 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.857063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.857067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.857462 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.857858 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.857865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.857868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.858000 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.858494 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:42.858498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:42.858645 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 16:58:42.858909 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 16:58:42.859414 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":123}
I20260619 16:58:42.863306 30096 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } attempt: 4
I20260619 16:58:42.865654 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:58:42.866255 30245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a: Bootstrap starting.
I20260619 16:58:42.870453 30245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:42.872041 30245 log.cc:824] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
I20260619 16:58:42.883188 30245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a: No bootstrap required, opened a new log
I20260619 16:58:42.888885 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.864599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.16.48.60:40179)
0619 16:58:42.864962 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.864967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.864986 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:58:42.865042 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.865046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.865121 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.865240 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.866214 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.867423 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.869644 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.870664 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.870834 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.880283 (+ 9449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.880291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.880296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.880344 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.884694 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.886029 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.886036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.886041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.886251 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.886783 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.886790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.886795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.887339 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.887506 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.888191 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.888454 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260619 16:58:42.891003 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.864294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60356)
0619 16:58:42.864527 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.864551 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:42.864682 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:42.865303 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:42.865308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:42.865835 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:42.866051 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.866064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.867833 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.867839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.869417 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.869436 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.870871 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.870877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.871104 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.871902 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:42.871927 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:42.878977 (+ 7050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:42.885154 (+ 6177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.885186 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.885190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.885849 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.886442 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.886447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.886451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.886632 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.887609 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:42.887614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:42.887836 (+ 222us) client_negotiation.cc:769] Sending connection context
0619 16:58:42.889980 (+ 2144us) client_negotiation.cc:240] Negotiation successful
0619 16:58:42.890543 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 16:58:42.898173 30096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850: Bootstrap starting.
I20260619 16:58:42.898852 30245 raft_consensus.cc:348] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } }
I20260619 16:58:42.900036 30245 raft_consensus.cc:374] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:42.900267 30245 raft_consensus.cc:729] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a209c19d91a84b3abe78e3ab7983a83a, State: Initialized, Role: FOLLOWER
I20260619 16:58:42.901285 30245 consensus_queue.cc:260] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } }
I20260619 16:58:42.903290 30096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:42.905887 30096 log.cc:824] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:42.905771 30262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:42.908951 30262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:58:42.908699 30245 sys_catalog.cc:564] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:58:42.910630 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.874530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.16.48.62:45313)
0619 16:58:42.874758 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.874764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.875432 (+ 668us) server_negotiation.cc:407] Connection header received
0619 16:58:42.875801 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.875810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.875918 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.876005 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.879796 (+ 3791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.880965 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.884070 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.884845 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.884944 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.888949 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.888957 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.888962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.889003 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.893865 (+ 4862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.906067 (+ 12202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.906074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.906079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.906287 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.908609 (+ 2322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.908615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.908619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.909030 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.909310 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.909966 (+ 656us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.910258 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:58:42.917893 30096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850: No bootstrap required, opened a new log
I20260619 16:58:42.920683 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.875526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.16.48.60:40179)
0619 16:58:42.889488 (+ 13962us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.889494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.889515 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:58:42.889573 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.889580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.889710 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.889815 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.898705 (+ 8890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.900015 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.902505 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.903105 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.903222 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.912058 (+ 8836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.912066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.912074 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.912152 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.916176 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.917426 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.917432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.917435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.917590 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.918065 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.918070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.918073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.918784 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.919027 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.919722 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.919969 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13836,"thread_start_us":15305,"threads_started":1}
I20260619 16:58:42.926455 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.891651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.16.48.61:45571)
0619 16:58:42.891877 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.891883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.902739 (+ 10856us) server_negotiation.cc:407] Connection header received
0619 16:58:42.903096 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.903102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.903198 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.903297 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.906344 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.907480 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.909673 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.910292 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.910395 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.914999 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.915012 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.915021 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.915155 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.920940 (+ 5785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.923463 (+ 2523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.923474 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.923480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.923711 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.924246 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.924253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.924259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.924724 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.925066 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.925835 (+ 769us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.926111 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:58:42.929164 30096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } }
I20260619 16:58:42.929548 30096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:42.929690 30096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72485882fcde48d286eed71c5200d850, State: Initialized, Role: FOLLOWER
I20260619 16:58:42.930388 30096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } }
I20260619 16:58:42.931347 30271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:42.931793 30271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:58:42.932884 30096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:58:42.939702 30275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:58:42.939859 30275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:58:42.935104 30178 master_runner.cc:186] Error getting master registration for 127.16.48.62:45313: OK, instance_id { permanent_uuid: "72485882fcde48d286eed71c5200d850" instance_seqno: 1781888321715484 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:58:42.946703 30178 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:58:42.953835 30287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:58:42.954016 30287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 16:58:42.954155 30050 tablet.cc:2369] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:58:42.956238 30029 master_runner.cc:186] Error getting master registration for 127.16.48.61:45571: OK, instance_id { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" instance_seqno: 1781888322240895 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:58:42.966727 30029 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:58:42.983626 30170 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } attempt: 5
Jun 19 16:58:42 dist-test-slave-3ffd krb5kdc[30016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888322, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
I20260619 16:58:42.999974 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.984399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:40179 (local address 127.0.0.1:60380)
0619 16:58:42.984664 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:42.984686 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:42.984853 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:42.985231 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:42.985236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:42.985724 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:42.985889 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.985897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.987002 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.987011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.988053 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:42.988066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.988889 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.988897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:42.989100 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.989781 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:42.989801 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:42.992008 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:42.997207 (+ 5199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.997214 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.997224 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.997750 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.998204 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:42.998209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:42.998211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:42.998351 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:42.998891 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:42.998897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:42.999053 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 16:58:42.999199 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:58:42.999540 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260619 16:58:42.999998 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:42.984454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.16.48.60:40179)
0619 16:58:42.984649 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:42.984655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:42.984771 (+ 116us) server_negotiation.cc:407] Connection header received
0619 16:58:42.984983 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:42.984990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:42.985061 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:42.985195 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:42.986048 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.986849 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.988242 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:42.988696 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:42.988843 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:42.992255 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:42.992260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:42.992263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:42.992299 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:42.997044 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.997902 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.997907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.997910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.998063 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:42.998474 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:42.998481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:42.998489 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:42.998765 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:42.998950 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:42.999294 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:58:42.999543 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:58:43.005338 30170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe: Bootstrap starting.
I20260619 16:58:43.010746 30170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:43.011879 30170 log.cc:824] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:43.015323 30170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe: No bootstrap required, opened a new log
W20260619 16:58:43.021350 30199 tablet.cc:2369] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:58:43.024636 30170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } }
I20260619 16:58:43.025089 30170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:43.025234 30170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326f6c1e2a514882b2f940ecf52a7dbe, State: Initialized, Role: FOLLOWER
I20260619 16:58:43.025915 30170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } }
I20260619 16:58:43.026891 30290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:43.027364 30290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:58:43.028283 30170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:58:43.042555 30262 raft_consensus.cc:482] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:58:43.043020 30262 raft_consensus.cc:504] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } }
I20260619 16:58:43.045315 30262 leader_election.cc:288] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72485882fcde48d286eed71c5200d850 (127.16.48.62:45313), 326f6c1e2a514882b2f940ecf52a7dbe (127.16.48.61:45571)
I20260619 16:58:43.046293 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a209c19d91a84b3abe78e3ab7983a83a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" is_pre_election: true
I20260619 16:58:43.046903 30145 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a209c19d91a84b3abe78e3ab7983a83a in term 0.
I20260619 16:58:43.047685 30197 leader_election.cc:302] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 326f6c1e2a514882b2f940ecf52a7dbe, a209c19d91a84b3abe78e3ab7983a83a; no voters:
I20260619 16:58:43.047824 30071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a209c19d91a84b3abe78e3ab7983a83a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72485882fcde48d286eed71c5200d850" is_pre_election: true
I20260619 16:58:43.048514 30071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a209c19d91a84b3abe78e3ab7983a83a in term 0.
I20260619 16:58:43.048528 30301 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:58:43.048956 30262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:58:43.049144 30262 raft_consensus.cc:482] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:58:43.049254 30262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:43.051637 30262 raft_consensus.cc:504] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } }
I20260619 16:58:43.053432 30071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a209c19d91a84b3abe78e3ab7983a83a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72485882fcde48d286eed71c5200d850"
I20260619 16:58:43.053717 30071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:43.057724 30071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a209c19d91a84b3abe78e3ab7983a83a in term 1.
I20260619 16:58:43.060037 30196 leader_election.cc:302] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72485882fcde48d286eed71c5200d850, a209c19d91a84b3abe78e3ab7983a83a; no voters:
I20260619 16:58:43.060875 30302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:58:43.068779 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a209c19d91a84b3abe78e3ab7983a83a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326f6c1e2a514882b2f940ecf52a7dbe"
I20260619 16:58:43.069095 30145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:43.077301 30145 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a209c19d91a84b3abe78e3ab7983a83a in term 1.
I20260619 16:58:43.080624 30262 leader_election.cc:288] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [CANDIDATE]: Term 1 election: Requested vote from peers 72485882fcde48d286eed71c5200d850 (127.16.48.62:45313), 326f6c1e2a514882b2f940ecf52a7dbe (127.16.48.61:45571)
I20260619 16:58:43.081527 30302 raft_consensus.cc:686] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [term 1 LEADER]: Becoming Leader. State: Replica: a209c19d91a84b3abe78e3ab7983a83a, State: Running, Role: LEADER
I20260619 16:58:43.082134 30302 consensus_queue.cc:237] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } }
I20260619 16:58:43.087497 30262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a209c19d91a84b3abe78e3ab7983a83a. Latest consensus state: current_term: 1 leader_uuid: "a209c19d91a84b3abe78e3ab7983a83a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:43.087836 30262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:43.088728 30305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:58:43.091499 30305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:58:43.100149 30145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [term 1 FOLLOWER]: Refusing update from remote peer a209c19d91a84b3abe78e3ab7983a83a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:58:43.100150 30071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [term 1 FOLLOWER]: Refusing update from remote peer a209c19d91a84b3abe78e3ab7983a83a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:58:43.101279 30262 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:58:43.101619 30302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [LEADER]: Connected to new peer: Peer: permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:58:43.125064 30290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a209c19d91a84b3abe78e3ab7983a83a. Latest consensus state: current_term: 1 leader_uuid: "a209c19d91a84b3abe78e3ab7983a83a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:43.125540 30290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:58:43.130280 30306 mvcc.cc:205] Tried to move back new op lower bound from 7298614571406807040 to 7298614571349422080. Current Snapshot: MvccSnapshot[applied={T|T < 7298614571406807040}]
I20260619 16:58:43.131341 30262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a209c19d91a84b3abe78e3ab7983a83a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:43.131671 30262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:43.135032 30305 catalog_manager.cc:1347] Generated new cluster ID: d5aa610479ec4e438065e027d28bf541
I20260619 16:58:43.135161 30305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:58:43.139822 30271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a209c19d91a84b3abe78e3ab7983a83a. Latest consensus state: current_term: 1 leader_uuid: "a209c19d91a84b3abe78e3ab7983a83a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:43.140213 30271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:58:43.140386 30303 sys_catalog.cc:455] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a209c19d91a84b3abe78e3ab7983a83a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:43.140841 30303 sys_catalog.cc:458] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:43.151492 30308 mvcc.cc:205] Tried to move back new op lower bound from 7298614571406807040 to 7298614571349422080. Current Snapshot: MvccSnapshot[applied={T|T < 7298614571406807040}]
I20260619 16:58:43.152837 30271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a209c19d91a84b3abe78e3ab7983a83a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:43.153141 30271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:58:43.161803 30305 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:58:43.163149 30290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a209c19d91a84b3abe78e3ab7983a83a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72485882fcde48d286eed71c5200d850" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45313 } } peers { permanent_uuid: "326f6c1e2a514882b2f940ecf52a7dbe" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 45571 } } peers { permanent_uuid: "a209c19d91a84b3abe78e3ab7983a83a" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 40179 } } }
I20260619 16:58:43.163436 30290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:58:43.163815 30305 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:58:43.183369 30305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a209c19d91a84b3abe78e3ab7983a83a: Generated new TSK 0
I20260619 16:58:43.184659 30305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:58:43.956097 30287 catalog_manager.cc:1259] Loaded cluster ID: d5aa610479ec4e438065e027d28bf541
I20260619 16:58:43.956326 30287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850: loading cluster ID for follower catalog manager: success
I20260619 16:58:43.959002 30287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850: acquiring CA information for follower catalog manager: success
I20260619 16:58:43.961000 30287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72485882fcde48d286eed71c5200d850: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:58:44.053025 30301 catalog_manager.cc:1259] Loaded cluster ID: d5aa610479ec4e438065e027d28bf541
I20260619 16:58:44.053202 30301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe: loading cluster ID for follower catalog manager: success
I20260619 16:58:44.055992 30301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe: acquiring CA information for follower catalog manager: success
I20260619 16:58:44.058146 30301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 326f6c1e2a514882b2f940ecf52a7dbe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:58:44.278736 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 30029
I20260619 16:58:44.280225 30091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:44.456403 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 30029
I20260619 16:58:44.478295 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 30103
I20260619 16:58:44.479571 30165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:44.585892 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 30103
I20260619 16:58:44.607585 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 30178
I20260619 16:58:44.608773 30240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:44.722803 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 30178
2026-06-19T16:58:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4867 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:58:44 dist-test-slave-3ffd krb5kdc[30325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:58:44 dist-test-slave-3ffd krb5kdc[30325](info): set up 2 sockets
Jun 19 16:58:44 dist-test-slave-3ffd krb5kdc[30325](info): commencing operation
krb5kdc: starting...
W20260619 16:58:45.832964 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:58:45 dist-test-slave-3ffd krb5kdc[30325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:58:46.083781 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:45593
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:45593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:46.431526 30338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:46.431923 30338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:46.432091 30338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:46.439594 30338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:58:46.439796 30338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:46.439915 30338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:46.440019 30338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:58:46.440088 30338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:58:46.450685 30338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:45593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:45593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:46.452184 30338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:46.453599 30338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:46.466216 30343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:46.466262 30346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:46.466286 30344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:46.467774 30338 server_base.cc:1034] running on GCE node
I20260619 16:58:46.468712 30338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:46.470300 30338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:46.471536 30338 hybrid_clock.cc:648] HybridClock initialized: now 1781888326471496 us; error 35 us; skew 500 ppm
I20260619 16:58:46.471884 30338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:46 dist-test-slave-3ffd krb5kdc[30325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888326, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:46.475297 30338 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:58:46.477125 30338 webserver.cc:460] Webserver started at http://127.16.48.62:43775/ using document root <none> and password file <none>
I20260619 16:58:46.477864 30338 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:46.477983 30338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:46.478271 30338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:58:46.481601 30338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "5c82eb44fdf7422b9584e28ef3129117"
format_stamp: "Formatted at 2026-06-19 16:58:46 on dist-test-slave-3ffd"
server_key: "c06538660243e3c62fea503f0dac63c1"
server_key_iv: "4be0f3504a90f6cd28790236798039c1"
server_key_version: "encryptionkey@0"
I20260619 16:58:46.482441 30338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "5c82eb44fdf7422b9584e28ef3129117"
format_stamp: "Formatted at 2026-06-19 16:58:46 on dist-test-slave-3ffd"
server_key: "c06538660243e3c62fea503f0dac63c1"
server_key_iv: "4be0f3504a90f6cd28790236798039c1"
server_key_version: "encryptionkey@0"
I20260619 16:58:46.488775 30338 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 16:58:46.492836 30353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:46.494432 30338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:58:46.494637 30338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "5c82eb44fdf7422b9584e28ef3129117"
format_stamp: "Formatted at 2026-06-19 16:58:46 on dist-test-slave-3ffd"
server_key: "c06538660243e3c62fea503f0dac63c1"
server_key_iv: "4be0f3504a90f6cd28790236798039c1"
server_key_version: "encryptionkey@0"
I20260619 16:58:46.494848 30338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:46.512425 30338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:46.517105 30338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:46.517376 30338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:46.537433 30338 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:45593
I20260619 16:58:46.537452 30404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:45593 every 8 connection(s)
I20260619 16:58:46.539041 30338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:58:46.543160 30405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:46.548745 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 30338
I20260619 16:58:46.549001 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:58:46.549667 16576 external_mini_cluster.cc:1428] Setting key ea4f124c2869c9ec05c07a15278649eb
I20260619 16:58:46.555122 30405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117: Bootstrap starting.
I20260619 16:58:46.559432 30405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:46.560942 30405 log.cc:824] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:46.564929 30405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117: No bootstrap required, opened a new log
Jun 19 16:58:46 dist-test-slave-3ffd krb5kdc[30325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:46.573822 30405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c82eb44fdf7422b9584e28ef3129117" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45593 } }
I20260619 16:58:46.574265 30405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:46.574397 30405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c82eb44fdf7422b9584e28ef3129117, State: Initialized, Role: FOLLOWER
I20260619 16:58:46.575170 30405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c82eb44fdf7422b9584e28ef3129117" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45593 } }
I20260619 16:58:46.575479 30405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:58:46.575635 30405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:58:46.575852 30405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:46.576792 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:46.551777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.16.48.62:45593)
0619 16:58:46.552759 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:46.552773 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:46.552824 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:58:46.554697 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:46.554730 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:46.555197 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:46.555660 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:46.557770 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:46.559163 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:46.561282 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:46.562507 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:46.562670 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:46.567851 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:46.567860 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:46.567878 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:46.567930 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:46.571367 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:46.572317 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:46.572326 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:46.572333 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:46.572536 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:46.573036 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:46.573046 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:46.573048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:46.573691 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:46.574046 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:46.574644 (+ 598us) server_negotiation.cc:299] Negotiation successful
0619 16:58:46.575174 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":488,"threads_started":1}
I20260619 16:58:46.578985 30405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c82eb44fdf7422b9584e28ef3129117" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45593 } }
I20260619 16:58:46.579684 30405 leader_election.cc:302] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c82eb44fdf7422b9584e28ef3129117; no voters:
I20260619 16:58:46.580422 30405 leader_election.cc:288] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:58:46.580576 30410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:58:46.581836 30410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [term 1 LEADER]: Becoming Leader. State: Replica: 5c82eb44fdf7422b9584e28ef3129117, State: Running, Role: LEADER
I20260619 16:58:46.581972 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:58:46.582720 30405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:58:46.582620 30410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c82eb44fdf7422b9584e28ef3129117" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45593 } }
I20260619 16:58:46.586226 30412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c82eb44fdf7422b9584e28ef3129117. Latest consensus state: current_term: 1 leader_uuid: "5c82eb44fdf7422b9584e28ef3129117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c82eb44fdf7422b9584e28ef3129117" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45593 } } }
I20260619 16:58:46.586732 30412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:46.587456 30411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c82eb44fdf7422b9584e28ef3129117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c82eb44fdf7422b9584e28ef3129117" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 45593 } } }
I20260619 16:58:46.587795 30411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:46.587792 30418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:58:46.595520 30418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:58:46.606060 30418 catalog_manager.cc:1347] Generated new cluster ID: 7091f9a0fd0e4700a69c6095191d3b2e
I20260619 16:58:46.606230 30418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:58:46.627756 30418 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:58:46.629338 30418 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:58:46.638192 30418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c82eb44fdf7422b9584e28ef3129117: Generated new TSK 0
I20260619 16:58:46.639204 30418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:58:46.696288 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 30338
I20260619 16:58:46.697857 30400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:46.802243 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 30338
2026-06-19T16:58:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2075 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:58:46 dist-test-slave-3ffd krb5kdc[30434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:58:46 dist-test-slave-3ffd krb5kdc[30434](info): set up 2 sockets
Jun 19 16:58:46 dist-test-slave-3ffd krb5kdc[30434](info): commencing operation
krb5kdc: starting...
W20260619 16:58:47.916941 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:58:48 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:58:48.173405 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42315
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:42315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:48.527048 30447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:48.527415 30447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:48.527583 30447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:48.535164 30447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:58:48.535357 30447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:48.535470 30447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:48.535578 30447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:58:48.535670 30447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:58:48.546046 30447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:42315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:48.547667 30447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:48.549149 30447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:48.562194 30453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:48.562234 30455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:48.562299 30452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:48.563069 30447 server_base.cc:1034] running on GCE node
I20260619 16:58:48.563992 30447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:48.565716 30447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:48.566946 30447 hybrid_clock.cc:648] HybridClock initialized: now 1781888328566896 us; error 42 us; skew 500 ppm
I20260619 16:58:48.567292 30447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:48 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888328, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:48.570644 30447 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:58:48.572330 30447 webserver.cc:460] Webserver started at http://127.16.48.62:45707/ using document root <none> and password file <none>
I20260619 16:58:48.573082 30447 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:48.573207 30447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:48.573520 30447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:58:48.576887 30447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "76b10fd3668f46ff82948228dfcd13c4"
format_stamp: "Formatted at 2026-06-19 16:58:48 on dist-test-slave-3ffd"
server_key: "753d8fc86897ff57db7be198b86403ec"
server_key_iv: "ded441724c4de97a59d060e428f53e58"
server_key_version: "encryptionkey@0"
I20260619 16:58:48.577713 30447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "76b10fd3668f46ff82948228dfcd13c4"
format_stamp: "Formatted at 2026-06-19 16:58:48 on dist-test-slave-3ffd"
server_key: "753d8fc86897ff57db7be198b86403ec"
server_key_iv: "ded441724c4de97a59d060e428f53e58"
server_key_version: "encryptionkey@0"
I20260619 16:58:48.583635 30447 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 16:58:48.587860 30462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:48.589660 30447 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:58:48.589918 30447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "76b10fd3668f46ff82948228dfcd13c4"
format_stamp: "Formatted at 2026-06-19 16:58:48 on dist-test-slave-3ffd"
server_key: "753d8fc86897ff57db7be198b86403ec"
server_key_iv: "ded441724c4de97a59d060e428f53e58"
server_key_version: "encryptionkey@0"
I20260619 16:58:48.590170 30447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:48.611759 30447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:48.616499 30447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:48.616850 30447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:48.637003 30447 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:42315
I20260619 16:58:48.637046 30513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:42315 every 8 connection(s)
I20260619 16:58:48.638484 30447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:58:48.639096 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 30447
I20260619 16:58:48.639307 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:58:48.640013 16576 external_mini_cluster.cc:1428] Setting key 5f17a5e242bdd57df151cbb2924e29c6
I20260619 16:58:48.643478 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:58:48 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:48.658273 30514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Bootstrap starting.
I20260619 16:58:48.662817 30514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:48.664273 30514 log.cc:824] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:48.667050 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:48.642329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.16.48.62:42315)
0619 16:58:48.643179 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:48.643190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:48.643233 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:58:48.645067 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:48.645091 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:48.645534 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:48.645896 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:48.647810 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:48.649431 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:48.651438 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:48.652268 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:48.652418 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:48.657487 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:48.657499 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:48.657524 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:48.657588 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:48.661842 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:48.662794 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:48.662810 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:48.662820 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:48.663050 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:48.663513 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:48.663517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:48.663520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:48.664361 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:48.664685 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:48.665187 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 16:58:48.665651 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":388,"threads_started":1}
I20260619 16:58:48.668074 30514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: No bootstrap required, opened a new log
I20260619 16:58:48.676306 30514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:48.676753 30514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:48.676867 30514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b10fd3668f46ff82948228dfcd13c4, State: Initialized, Role: FOLLOWER
I20260619 16:58:48.677675 30514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:48.677992 30514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:58:48.678148 30514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:58:48.678342 30514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:48.680714 30514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:48.681283 30514 leader_election.cc:302] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b10fd3668f46ff82948228dfcd13c4; no voters:
I20260619 16:58:48.681916 30514 leader_election.cc:288] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:58:48.682083 30519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:58:48.682821 30519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 1 LEADER]: Becoming Leader. State: Replica: 76b10fd3668f46ff82948228dfcd13c4, State: Running, Role: LEADER
I20260619 16:58:48.683411 30519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:48.684290 30514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:58:48.686401 30521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b10fd3668f46ff82948228dfcd13c4. Latest consensus state: current_term: 1 leader_uuid: "76b10fd3668f46ff82948228dfcd13c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } } }
I20260619 16:58:48.686846 30521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:48.688362 30520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76b10fd3668f46ff82948228dfcd13c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } } }
I20260619 16:58:48.688786 30520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:48.690105 30527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:58:48.695755 30527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:58:48.707655 30536 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:58:48.713244 30527 catalog_manager.cc:1347] Generated new cluster ID: 9f6685d7f46349cdbae402ec4a937484
I20260619 16:58:48.713465 30527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:58:48.729909 30527 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:58:48.731367 30527 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:58:48.738067 30527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Generated new TSK 0
I20260619 16:58:48.739228 30527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:58:48.812065 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42315
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:49.169023 30540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:49.169392 30540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:49.169566 30540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:49.177342 30540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:49.177511 30540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:49.177693 30540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:58:49.188457 30540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:49.190037 30540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:49.191502 30540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:49.206581 30546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:49.206466 30545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:49.207032 30548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:49.207851 30540 server_base.cc:1034] running on GCE node
I20260619 16:58:49.208662 30540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:49.209738 30540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:49.210990 30540 hybrid_clock.cc:648] HybridClock initialized: now 1781888329210913 us; error 69 us; skew 500 ppm
I20260619 16:58:49.211391 30540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:49 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888329, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:49.215168 30540 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:58:49.216961 30540 webserver.cc:460] Webserver started at http://127.16.48.1:34435/ using document root <none> and password file <none>
I20260619 16:58:49.217743 30540 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:49.217888 30540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:49.218225 30540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:58:49.221549 30540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "a99cb23811a041abb005505b6d958daf"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "b64d4274d6f4e4f9b88a73c18d5bb55c"
server_key_iv: "d87eb4298e3b6b7c6dd1096cedd9f2d1"
server_key_version: "encryptionkey@0"
I20260619 16:58:49.222491 30540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "a99cb23811a041abb005505b6d958daf"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "b64d4274d6f4e4f9b88a73c18d5bb55c"
server_key_iv: "d87eb4298e3b6b7c6dd1096cedd9f2d1"
server_key_version: "encryptionkey@0"
I20260619 16:58:49.228545 30540 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 16:58:49.232434 30555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:49.234010 30540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:58:49.234259 30540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "a99cb23811a041abb005505b6d958daf"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "b64d4274d6f4e4f9b88a73c18d5bb55c"
server_key_iv: "d87eb4298e3b6b7c6dd1096cedd9f2d1"
server_key_version: "encryptionkey@0"
I20260619 16:58:49.234505 30540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:49.253489 30540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:49.258029 30540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:49.258347 30540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:49.259373 30540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:49.260910 30540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:49.261015 30540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:49.261164 30540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:49.261233 30540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:49.295859 30540 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:42463
I20260619 16:58:49.295904 30667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:42463 every 8 connection(s)
I20260619 16:58:49.297317 30540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:58:49.299706 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 30540
I20260619 16:58:49.299952 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:58:49.300730 16576 external_mini_cluster.cc:1428] Setting key 9c67685efcdeced392a059eba7719f76
Jun 19 16:58:49 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888329, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:49.320875 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:49.300314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:37525 (local address 127.16.48.62:42315)
0619 16:58:49.300851 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:49.300859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:49.302040 (+ 1181us) server_negotiation.cc:407] Connection header received
0619 16:58:49.304066 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:49.304072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:49.304223 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:49.304356 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:49.306135 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:49.306974 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:49.308837 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:49.309271 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:49.309421 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:49.314023 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:49.314029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:49.314032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:49.314075 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:49.316948 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:49.318128 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:49.318133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:49.318135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:49.318273 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:49.318843 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:49.318846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:49.318848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:49.319137 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:49.319257 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:49.320374 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0619 16:58:49.320599 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
I20260619 16:58:49.322605 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:49.300975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:37525)
0619 16:58:49.301862 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:49.301913 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:49.303736 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:49.304666 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:49.304676 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:49.305271 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:49.305917 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:49.305938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:49.307204 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:49.307208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:49.308642 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:49.308658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:49.309542 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:49.309550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:49.309817 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:49.310630 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:49.310660 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:49.313702 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:49.317157 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:49.317171 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:49.317196 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:49.317949 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:49.318410 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:49.318418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:49.318425 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:49.318724 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:49.319281 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:49.319293 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:49.319834 (+ 541us) client_negotiation.cc:769] Sending connection context
0619 16:58:49.320191 (+ 357us) client_negotiation.cc:240] Negotiation successful
0619 16:58:49.320890 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":373,"threads_started":1}
I20260619 16:58:49.324424 30668 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42315
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 16:58:49.324846 30668 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:49.325700 30668 heartbeater.cc:511] Master 127.16.48.62:42315 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:58:49.328284 30479 ts_manager.cc:194] Registered new tserver with Master: a99cb23811a041abb005505b6d958daf (127.16.48.1:42463)
I20260619 16:58:49.330570 30479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:37525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:58:49.359159 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42315
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:49.717150 30674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:49.717542 30674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:49.717685 30674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:49.725250 30674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:49.725433 30674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:49.725620 30674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:58:49.735915 30674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:49.737378 30674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:49.738957 30674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:49.752631 30679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:49.752730 30680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:49.752839 30682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:49.753837 30674 server_base.cc:1034] running on GCE node
I20260619 16:58:49.754580 30674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:49.755594 30674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:49.756824 30674 hybrid_clock.cc:648] HybridClock initialized: now 1781888329756767 us; error 48 us; skew 500 ppm
I20260619 16:58:49.757211 30674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:49 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888329, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:49.760514 30674 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:58:49.762045 30674 webserver.cc:460] Webserver started at http://127.16.48.2:41073/ using document root <none> and password file <none>
I20260619 16:58:49.762800 30674 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:49.762913 30674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:49.763191 30674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:58:49.766366 30674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "fc55e6575ee148708d592d15a00d54ca"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "c4b987cea9fecad0360e16ec74753f06"
server_key_iv: "db4465676c87649b0ff8f9fba7736661"
server_key_version: "encryptionkey@0"
I20260619 16:58:49.767205 30674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "fc55e6575ee148708d592d15a00d54ca"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "c4b987cea9fecad0360e16ec74753f06"
server_key_iv: "db4465676c87649b0ff8f9fba7736661"
server_key_version: "encryptionkey@0"
I20260619 16:58:49.773034 30674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:58:49.777025 30689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:49.778735 30674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:58:49.778945 30674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "fc55e6575ee148708d592d15a00d54ca"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "c4b987cea9fecad0360e16ec74753f06"
server_key_iv: "db4465676c87649b0ff8f9fba7736661"
server_key_version: "encryptionkey@0"
I20260619 16:58:49.779207 30674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:49.811610 30674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:49.816381 30674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:49.816717 30674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:49.817876 30674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:49.819490 30674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:49.819612 30674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:49.819767 30674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:49.819837 30674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:49.855321 30674 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:42527
I20260619 16:58:49.855386 30801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:42527 every 8 connection(s)
I20260619 16:58:49.856961 30674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:58:49.864617 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 30674
I20260619 16:58:49.864825 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:58:49.865283 16576 external_mini_cluster.cc:1428] Setting key ee93ade483d4e0fa1c243cc65e5f152c
Jun 19 16:58:49 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888329, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:49.881305 30804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:49.860015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:56101 (local address 127.16.48.62:42315)
0619 16:58:49.860581 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:49.860586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:49.861712 (+ 1126us) server_negotiation.cc:407] Connection header received
0619 16:58:49.863766 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:49.863771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:49.863885 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:49.864014 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:49.866119 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:49.867008 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:49.869065 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:49.869533 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:49.869688 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:49.874150 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:49.874162 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:49.874170 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:49.874228 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:49.877258 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:49.878506 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:49.878511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:49.878513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:49.878671 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:49.879302 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:49.879306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:49.879308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:49.879635 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:49.879759 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:49.880812 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0619 16:58:49.881025 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":344,"threads_started":1}
I20260619 16:58:49.882979 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:49.860692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:56101)
0619 16:58:49.861503 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:49.861556 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:49.863402 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:49.864346 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:49.864357 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:49.865058 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:49.865829 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:49.865863 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:49.867278 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:49.867284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:49.868849 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:49.868866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:49.869806 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:49.869814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:49.870089 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:49.870872 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:49.870898 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:49.873751 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:49.877465 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:49.877481 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:49.877506 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:49.878283 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:49.878804 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:49.878812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:49.878819 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:49.879152 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:49.879774 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:49.879787 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:49.880346 (+ 559us) client_negotiation.cc:769] Sending connection context
0619 16:58:49.880676 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 16:58:49.881346 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":333,"threads_started":1}
I20260619 16:58:49.884883 30802 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42315
I20260619 16:58:49.885314 30802 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:49.886181 30802 heartbeater.cc:511] Master 127.16.48.62:42315 requested a full tablet report, sending...
I20260619 16:58:49.888093 30479 ts_manager.cc:194] Registered new tserver with Master: fc55e6575ee148708d592d15a00d54ca (127.16.48.2:42527)
I20260619 16:58:49.889118 30479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:56101
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:58:49.923782 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42315
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:50.273289 30808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:50.273679 30808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:50.273849 30808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:50.282018 30808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:50.282191 30808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:50.282393 30808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:58:50.292944 30808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:50.294346 30808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:50.295854 30808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:50.309958 30813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:50.309999 30814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:50.310039 30816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:50.311311 30808 server_base.cc:1034] running on GCE node
I20260619 16:58:50.311976 30808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:50.313014 30808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:50.314251 30808 hybrid_clock.cc:648] HybridClock initialized: now 1781888330314198 us; error 50 us; skew 500 ppm
I20260619 16:58:50.314591 30808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:50 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888330, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:50.318037 30808 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:58:50.319731 30808 webserver.cc:460] Webserver started at http://127.16.48.3:45849/ using document root <none> and password file <none>
I20260619 16:58:50.320590 30808 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:50.320744 30808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:50.321065 30808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:58:50.324368 30808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "571c05a4dbfb496ab8fb07213beda5d4"
format_stamp: "Formatted at 2026-06-19 16:58:50 on dist-test-slave-3ffd"
server_key: "09f9c51c88e368d4d528597078aafe72"
server_key_iv: "d38faa49c30f8e9d911bf89b169df19a"
server_key_version: "encryptionkey@0"
I20260619 16:58:50.325289 30808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "571c05a4dbfb496ab8fb07213beda5d4"
format_stamp: "Formatted at 2026-06-19 16:58:50 on dist-test-slave-3ffd"
server_key: "09f9c51c88e368d4d528597078aafe72"
server_key_iv: "d38faa49c30f8e9d911bf89b169df19a"
server_key_version: "encryptionkey@0"
I20260619 16:58:50.331069 30808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:58:50.334232 30668 heartbeater.cc:503] Master 127.16.48.62:42315 was elected leader, sending a full tablet report...
I20260619 16:58:50.335021 30823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:50.336732 30808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:58:50.336882 30808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "571c05a4dbfb496ab8fb07213beda5d4"
format_stamp: "Formatted at 2026-06-19 16:58:50 on dist-test-slave-3ffd"
server_key: "09f9c51c88e368d4d528597078aafe72"
server_key_iv: "d38faa49c30f8e9d911bf89b169df19a"
server_key_version: "encryptionkey@0"
I20260619 16:58:50.337073 30808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:50.352263 30808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:50.356982 30808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:50.357249 30808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:50.358322 30808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:50.359737 30808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:50.359813 30808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:50.359899 30808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:50.359946 30808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:50.395453 30808 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:45753
I20260619 16:58:50.395489 30935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:45753 every 8 connection(s)
I20260619 16:58:50.396848 30808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:58:50.399148 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 30808
I20260619 16:58:50.399364 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:58:50.399945 16576 external_mini_cluster.cc:1428] Setting key 23d3ef36a2c942feff02735a5280d458
Jun 19 16:58:50 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888330, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:50.418605 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.399503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:54747 (local address 127.16.48.62:42315)
0619 16:58:50.400022 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.400029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.401258 (+ 1229us) server_negotiation.cc:407] Connection header received
0619 16:58:50.403618 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.403622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.403706 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.403794 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:50.405570 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.406415 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.408074 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.408567 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.408713 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.412328 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:50.412334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:50.412337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:50.412375 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:50.415283 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:50.416235 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:50.416239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:50.416241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:50.416372 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:50.416818 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:50.416823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:50.416825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:50.417130 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:50.417264 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.418012 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.418226 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":241,"threads_started":1}
I20260619 16:58:50.419679 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.400215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:54747)
0619 16:58:50.401090 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:50.401148 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:50.403280 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:50.404024 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:50.404033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:50.404733 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:50.405375 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.405396 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.406602 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.406606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.407909 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.407922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.408826 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.408831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.409052 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.409814 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:50.409838 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:50.412040 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:50.415428 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:50.415437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:50.415456 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:50.416082 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:50.416467 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:50.416470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:50.416472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:50.416668 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:50.417251 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:50.417258 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:50.417633 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 16:58:50.417872 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 16:58:50.418382 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":339,"threads_started":1}
I20260619 16:58:50.421659 30936 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42315
I20260619 16:58:50.422034 30936 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:50.422888 30936 heartbeater.cc:511] Master 127.16.48.62:42315 requested a full tablet report, sending...
I20260619 16:58:50.424624 30479 ts_manager.cc:194] Registered new tserver with Master: 571c05a4dbfb496ab8fb07213beda5d4 (127.16.48.3:45753)
I20260619 16:58:50.425518 30479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:54747
I20260619 16:58:50.428668 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:58:50.447863 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.433619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.16.48.62:42315)
0619 16:58:50.433824 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.433829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.434162 (+ 333us) server_negotiation.cc:407] Connection header received
0619 16:58:50.434324 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.434328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.434399 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.434503 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:50.436014 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.436880 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.438344 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.438770 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.438877 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.441690 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:50.441695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:50.441699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:50.441740 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:50.444632 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:50.445509 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:50.445516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:50.445520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:50.445735 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:50.446210 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:50.446216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:50.446219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:50.446600 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:50.446915 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.447258 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.447514 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:58:50.452332 30479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:58:50.459512 30479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:58:50.489465 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.472384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:42527 (local address 127.0.0.1:48826)
0619 16:58:50.473960 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:50.474022 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:50.474238 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:50.477657 (+ 3419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:50.477669 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:50.477716 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:50.478199 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.478215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.482292 (+ 4077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.482302 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.486349 (+ 4047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.486364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.488241 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.488248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.488476 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.488581 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 16:58:50.488766 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 16:58:50.489017 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235,"thread_start_us":11767,"threads_started":1}
I20260619 16:58:50.490207 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.472597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.16.48.2:42527)
0619 16:58:50.474970 (+ 2373us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.474983 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.475017 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:58:50.475143 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.475156 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.477002 (+ 1846us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.477262 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:50.478463 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.482079 (+ 3616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.486559 (+ 4480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.487991 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.488202 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.488434 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.488823 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.489018 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2153,"thread_start_us":1993,"threads_started":1}
I20260619 16:58:50.492918 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.472588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.0.0.1:48526)
0619 16:58:50.483427 (+ 10839us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:50.483452 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:50.483615 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:50.485194 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:50.485205 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:50.485231 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:50.485610 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.485622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.487591 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.487597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.489923 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.489941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.491967 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.491974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.492199 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.492235 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:58:50.492336 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:58:50.492496 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10667,"mutex_wait_us":21,"thread_start_us":914,"threads_started":1}
I20260619 16:58:50.492937 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.473255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.16.48.1:42463)
0619 16:58:50.483853 (+ 10598us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.483862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.483894 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:58:50.484180 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.484192 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.484531 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.484881 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:50.485876 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.487453 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.490404 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.491748 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.491893 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.492165 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.492399 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.492578 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10378,"thread_start_us":10436,"threads_started":1}
I20260619 16:58:50.493809 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.475858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:45753 (local address 127.0.0.1:57696)
0619 16:58:50.484412 (+ 8554us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:50.484435 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:50.484547 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:50.486378 (+ 1831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:50.486383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:50.486409 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:50.486769 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.486779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.488668 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.488673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.490942 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.490956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.493089 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.493095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.493237 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.493260 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:58:50.493372 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:58:50.493529 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8420,"mutex_wait_us":28,"thread_start_us":7219,"threads_started":1}
I20260619 16:58:50.494623 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.476537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.16.48.3:45753)
0619 16:58:50.485850 (+ 9313us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.485859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.485895 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:58:50.485984 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.485991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.486189 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.486395 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:50.486944 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.488526 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.491126 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.492580 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.492718 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.492945 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.494107 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.494247 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9138,"thread_start_us":8986,"threads_started":1}
I20260619 16:58:50.494789 30737 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5ba07020c24d78ac380d27cb71a397 (DEFAULT_TABLE table=test-table [id=177e6d75e8b54d85ae74593601e0c877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:58:50.496527 30737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5ba07020c24d78ac380d27cb71a397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:50.497287 30603 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5ba07020c24d78ac380d27cb71a397 (DEFAULT_TABLE table=test-table [id=177e6d75e8b54d85ae74593601e0c877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:58:50.498558 30603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5ba07020c24d78ac380d27cb71a397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:50.498688 30871 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5ba07020c24d78ac380d27cb71a397 (DEFAULT_TABLE table=test-table [id=177e6d75e8b54d85ae74593601e0c877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:58:50.500209 30871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5ba07020c24d78ac380d27cb71a397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:50.510888 30955 tablet_bootstrap.cc:492] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: Bootstrap starting.
I20260619 16:58:50.513236 30956 tablet_bootstrap.cc:492] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: Bootstrap starting.
I20260619 16:58:50.516544 30955 tablet_bootstrap.cc:654] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:50.516907 30957 tablet_bootstrap.cc:492] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: Bootstrap starting.
I20260619 16:58:50.516983 30956 tablet_bootstrap.cc:654] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:50.518451 30956 log.cc:824] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:50.518447 30955 log.cc:824] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:50.520479 30957 tablet_bootstrap.cc:654] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:50.521643 30955 tablet_bootstrap.cc:492] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: No bootstrap required, opened a new log
I20260619 16:58:50.521879 30957 log.cc:824] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:50.521940 30955 ts_tablet_manager.cc:1397] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 16:58:50.522019 30956 tablet_bootstrap.cc:492] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: No bootstrap required, opened a new log
I20260619 16:58:50.522341 30956 ts_tablet_manager.cc:1397] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.007s
I20260619 16:58:50.525810 30957 tablet_bootstrap.cc:492] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: No bootstrap required, opened a new log
I20260619 16:58:50.526105 30957 ts_tablet_manager.cc:1397] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 16:58:50.530248 30956 raft_consensus.cc:348] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } }
I20260619 16:58:50.530706 30956 raft_consensus.cc:374] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:50.530830 30956 raft_consensus.cc:729] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc55e6575ee148708d592d15a00d54ca, State: Initialized, Role: FOLLOWER
I20260619 16:58:50.531548 30956 consensus_queue.cc:260] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } }
I20260619 16:58:50.532809 30802 heartbeater.cc:503] Master 127.16.48.62:42315 was elected leader, sending a full tablet report...
I20260619 16:58:50.532896 30955 raft_consensus.cc:348] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } }
I20260619 16:58:50.533409 30955 raft_consensus.cc:374] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:50.533509 30955 raft_consensus.cc:729] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99cb23811a041abb005505b6d958daf, State: Initialized, Role: FOLLOWER
I20260619 16:58:50.533600 30956 ts_tablet_manager.cc:1428] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: Time spent starting tablet: real 0.011s user 0.000s sys 0.010s
I20260619 16:58:50.534307 30955 consensus_queue.cc:260] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } }
I20260619 16:58:50.536360 30955 ts_tablet_manager.cc:1428] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260619 16:58:50.538400 30957 raft_consensus.cc:348] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } }
I20260619 16:58:50.538930 30957 raft_consensus.cc:374] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:50.539040 30957 raft_consensus.cc:729] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571c05a4dbfb496ab8fb07213beda5d4, State: Initialized, Role: FOLLOWER
I20260619 16:58:50.539737 30957 consensus_queue.cc:260] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } }
I20260619 16:58:50.541023 30936 heartbeater.cc:503] Master 127.16.48.62:42315 was elected leader, sending a full tablet report...
I20260619 16:58:50.541712 30957 ts_tablet_manager.cc:1428] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260619 16:58:50.549796 30669 tablet.cc:2369] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:58:50.609217 30803 tablet.cc:2369] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:58:50.643021 30961 raft_consensus.cc:482] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:58:50.643390 30961 raft_consensus.cc:504] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } }
I20260619 16:58:50.645325 30961 leader_election.cc:288] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a99cb23811a041abb005505b6d958daf (127.16.48.1:42463), 571c05a4dbfb496ab8fb07213beda5d4 (127.16.48.3:45753)
W20260619 16:58:50.648451 30937 tablet.cc:2369] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:58:50.654019 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.646040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:53745 (local address 127.16.48.3:45753)
0619 16:58:50.646275 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.646284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.646860 (+ 576us) server_negotiation.cc:407] Connection header received
0619 16:58:50.647172 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.647177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.647261 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.647311 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:50.648346 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.649509 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.652009 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.652923 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.653119 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.653264 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.653602 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.653734 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:58:50.654212 30964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.645593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:45753 (local address 127.16.48.2:53745)
0619 16:58:50.646723 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:50.646748 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:50.646988 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:50.647507 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:50.647510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:50.647593 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:50.648190 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.648201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.649664 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.649670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.651818 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.651834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.653270 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.653278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.653427 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.653453 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:58:50.653576 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:58:50.653733 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":41,"spinlock_wait_cycles":25344,"thread_start_us":706,"threads_started":1}
I20260619 16:58:50.655368 30891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5ba07020c24d78ac380d27cb71a397" candidate_uuid: "fc55e6575ee148708d592d15a00d54ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571c05a4dbfb496ab8fb07213beda5d4" is_pre_election: true
I20260619 16:58:50.655905 30891 raft_consensus.cc:2454] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc55e6575ee148708d592d15a00d54ca in term 0.
I20260619 16:58:50.656347 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.645855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:51301 (local address 127.16.48.1:42463)
0619 16:58:50.646055 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.646062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.647099 (+ 1037us) server_negotiation.cc:407] Connection header received
0619 16:58:50.647265 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.647270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.647351 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.647443 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:50.648348 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.649497 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.654266 (+ 4769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.655383 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.655551 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.655664 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.655950 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.656079 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:58:50.656495 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.645593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.16.48.2:51301)
0619 16:58:50.646985 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:50.647001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:50.647115 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:50.647492 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:50.647496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:50.647523 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:50.648097 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.648150 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.651995 (+ 3845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.652000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.654037 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:50.654054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.655601 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.655607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:50.655785 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.655809 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:58:50.655923 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:58:50.656061 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"mutex_wait_us":509,"thread_start_us":548,"threads_started":1}
I20260619 16:58:50.656697 30691 leader_election.cc:302] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571c05a4dbfb496ab8fb07213beda5d4, fc55e6575ee148708d592d15a00d54ca; no voters:
I20260619 16:58:50.657166 30961 raft_consensus.cc:2781] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:58:50.657305 30961 raft_consensus.cc:482] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:58:50.657420 30961 raft_consensus.cc:3037] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:50.657635 30623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5ba07020c24d78ac380d27cb71a397" candidate_uuid: "fc55e6575ee148708d592d15a00d54ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99cb23811a041abb005505b6d958daf" is_pre_election: true
I20260619 16:58:50.658118 30623 raft_consensus.cc:2454] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc55e6575ee148708d592d15a00d54ca in term 0.
I20260619 16:58:50.659983 30961 raft_consensus.cc:504] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } }
I20260619 16:58:50.660734 30961 leader_election.cc:288] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [CANDIDATE]: Term 1 election: Requested vote from peers a99cb23811a041abb005505b6d958daf (127.16.48.1:42463), 571c05a4dbfb496ab8fb07213beda5d4 (127.16.48.3:45753)
I20260619 16:58:50.661180 30623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5ba07020c24d78ac380d27cb71a397" candidate_uuid: "fc55e6575ee148708d592d15a00d54ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99cb23811a041abb005505b6d958daf"
I20260619 16:58:50.661332 30891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5ba07020c24d78ac380d27cb71a397" candidate_uuid: "fc55e6575ee148708d592d15a00d54ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571c05a4dbfb496ab8fb07213beda5d4"
I20260619 16:58:50.661422 30623 raft_consensus.cc:3037] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:50.661571 30891 raft_consensus.cc:3037] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:50.664855 30891 raft_consensus.cc:2454] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc55e6575ee148708d592d15a00d54ca in term 1.
I20260619 16:58:50.664855 30623 raft_consensus.cc:2454] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc55e6575ee148708d592d15a00d54ca in term 1.
I20260619 16:58:50.665489 30691 leader_election.cc:302] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 571c05a4dbfb496ab8fb07213beda5d4, a99cb23811a041abb005505b6d958daf, fc55e6575ee148708d592d15a00d54ca; no voters:
I20260619 16:58:50.665884 30961 raft_consensus.cc:2781] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:58:50.666399 30961 raft_consensus.cc:686] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 1 LEADER]: Becoming Leader. State: Replica: fc55e6575ee148708d592d15a00d54ca, State: Running, Role: LEADER
I20260619 16:58:50.666914 30961 consensus_queue.cc:237] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } }
I20260619 16:58:50.672264 30479 catalog_manager.cc:5495] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca reported cstate change: term changed from 0 to 1, leader changed from <none> to fc55e6575ee148708d592d15a00d54ca (127.16.48.2). New cstate: current_term: 1 leader_uuid: "fc55e6575ee148708d592d15a00d54ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } health_report { overall_health: UNKNOWN } } }
I20260619 16:58:50.724232 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.718494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.16.48.2:42527)
0619 16:58:50.718752 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.718756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.718775 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:58:50.718963 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.718967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.719060 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.719211 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:58:50.719554 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.720362 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.721809 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.722209 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.722307 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.722964 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:58:50.723524 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:58:50.723642 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.723776 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.723852 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:58:50.733539 30891 raft_consensus.cc:1261] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [term 1 FOLLOWER]: Refusing update from remote peer fc55e6575ee148708d592d15a00d54ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:58:50.733539 30623 raft_consensus.cc:1261] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [term 1 FOLLOWER]: Refusing update from remote peer fc55e6575ee148708d592d15a00d54ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:58:50.734719 30966 consensus_queue.cc:1035] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:58:50.735131 30961 consensus_queue.cc:1035] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:58:50.746438 30969 mvcc.cc:205] Tried to move back new op lower bound from 7298614602672635904 to 7298614602415656960. Current Snapshot: MvccSnapshot[applied={T|T < 7298614602672635904}]
I20260619 16:58:50.752393 30971 mvcc.cc:205] Tried to move back new op lower bound from 7298614602672635904 to 7298614602415656960. Current Snapshot: MvccSnapshot[applied={T|T < 7298614602672635904}]
I20260619 16:58:50.763195 30478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:58:50.763543 30478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:58:50.767743 30478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Sending DeleteTablet for 3 replicas of tablet 9c5ba07020c24d78ac380d27cb71a397
I20260619 16:58:50.769088 30603 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5ba07020c24d78ac380d27cb71a397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:58:50 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:48526
I20260619 16:58:50.769162 30871 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5ba07020c24d78ac380d27cb71a397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:58:50 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:57696
I20260619 16:58:50.769819 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 30540
I20260619 16:58:50.769953 30737 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5ba07020c24d78ac380d27cb71a397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:58:50 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:48826
I20260619 16:58:50.781410 30979 tablet_replica.cc:330] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: stopping tablet replica
I20260619 16:58:50.781950 30978 tablet_replica.cc:330] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: stopping tablet replica
I20260619 16:58:50.782124 30980 tablet_replica.cc:330] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: stopping tablet replica
I20260619 16:58:50.782626 30978 raft_consensus.cc:2229] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:58:50.782594 30980 raft_consensus.cc:2229] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:58:50.783355 30978 raft_consensus.cc:2258] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:58:50.783979 30979 raft_consensus.cc:2229] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:58:50.784801 30979 raft_consensus.cc:2258] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:58:50.783345 30980 raft_consensus.cc:2258] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:58:50.786374 30978 ts_tablet_manager.cc:1905] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:58:50.787964 30980 ts_tablet_manager.cc:1905] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:58:50.789578 30979 ts_tablet_manager.cc:1905] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:58:50 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:58:50.794636 30978 ts_tablet_manager.cc:1918] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:58:50.795054 30978 log.cc:1196] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/9c5ba07020c24d78ac380d27cb71a397
I20260619 16:58:50.795742 30978 ts_tablet_manager.cc:1939] T 9c5ba07020c24d78ac380d27cb71a397 P a99cb23811a041abb005505b6d958daf: Deleting consensus metadata
I20260619 16:58:50.798025 30466 catalog_manager.cc:4841] TS a99cb23811a041abb005505b6d958daf (127.16.48.1:42463): tablet 9c5ba07020c24d78ac380d27cb71a397 (table test-table [id=177e6d75e8b54d85ae74593601e0c877]) successfully deleted
I20260619 16:58:50.801548 30979 ts_tablet_manager.cc:1918] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:58:50.801872 30979 log.cc:1196] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/9c5ba07020c24d78ac380d27cb71a397
I20260619 16:58:50.802562 30979 ts_tablet_manager.cc:1939] T 9c5ba07020c24d78ac380d27cb71a397 P 571c05a4dbfb496ab8fb07213beda5d4: Deleting consensus metadata
I20260619 16:58:50.802857 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.771388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.16.48.1:42463)
0619 16:58:50.771589 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.771607 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.780354 (+ 8747us) server_negotiation.cc:407] Connection header received
0619 16:58:50.780614 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.780621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.780757 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.780876 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:50.783284 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.784649 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.786931 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.787629 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.787753 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.794010 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:50.794037 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:50.794066 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:50.794135 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:50.798513 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:50.799568 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:50.799578 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:50.799582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:50.799782 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:50.800345 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:50.800350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:50.800354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:50.800856 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:50.801285 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.802050 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.802377 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:58:50.804502 30663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:50.805198 30464 catalog_manager.cc:4841] TS 571c05a4dbfb496ab8fb07213beda5d4 (127.16.48.3:45753): tablet 9c5ba07020c24d78ac380d27cb71a397 (table test-table [id=177e6d75e8b54d85ae74593601e0c877]) successfully deleted
I20260619 16:58:50.806578 30980 ts_tablet_manager.cc:1918] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:58:50.806900 30980 log.cc:1196] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/9c5ba07020c24d78ac380d27cb71a397
I20260619 16:58:50.807786 30980 ts_tablet_manager.cc:1939] T 9c5ba07020c24d78ac380d27cb71a397 P fc55e6575ee148708d592d15a00d54ca: Deleting consensus metadata
I20260619 16:58:50.809991 30463 catalog_manager.cc:4841] TS fc55e6575ee148708d592d15a00d54ca (127.16.48.2:42527): tablet 9c5ba07020c24d78ac380d27cb71a397 (table test-table [id=177e6d75e8b54d85ae74593601e0c877]) successfully deleted
I20260619 16:58:50.911798 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 30540
I20260619 16:58:50.937501 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 30674
Jun 19 16:58:50 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:58:50.957046 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:50.938478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.16.48.2:42527)
0619 16:58:50.938678 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:50.938682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:50.938704 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:58:50.938822 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:50.938826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:50.938904 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:50.938990 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:50.940940 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.941963 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.943992 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:50.944565 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:50.944706 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:50.950323 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:50.950341 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:50.950358 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:50.950421 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:50.953679 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:50.954679 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:50.954688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:50.954690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:50.954869 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:50.955356 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:50.955359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:50.955362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:50.955793 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:50.956075 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:50.956515 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 16:58:50.956708 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:58:50.957906 30797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:51.075395 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 30674
I20260619 16:58:51.095201 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 30808
Jun 19 16:58:51 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:58:51.112653 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:51.096028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.16.48.3:45753)
0619 16:58:51.096250 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:51.096258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:51.096282 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:58:51.096505 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:51.096511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:51.096605 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:51.096718 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:51.098146 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:51.098886 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:51.100372 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:51.100798 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:51.100952 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:51.105773 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:51.105793 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:51.105810 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:51.105882 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:51.109284 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:51.110174 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:51.110183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:51.110186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:51.110340 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:51.110776 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:51.110780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:51.110782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:51.111276 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:51.111617 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:51.111999 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 16:58:51.112283 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:58:51.113371 30931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:51.204816 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 30808
I20260619 16:58:51.222525 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 30447
I20260619 16:58:51.223848 30509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:51.339663 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 30447
I20260619 16:58:51.362596 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42315
--webserver_interface=127.16.48.62
--webserver_port=45707
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:42315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:51.718858 30986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:51.719178 30986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:51.719365 30986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:51.726873 30986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:58:51.727030 30986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:51.727121 30986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:51.727210 30986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:58:51.727288 30986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:58:51.737673 30986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:42315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=45707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:51.739077 30986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:51.740594 30986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:51.764396 30992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:51.764427 30995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:51.764402 30993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:51.765898 30986 server_base.cc:1034] running on GCE node
I20260619 16:58:51.766966 30986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:51.768838 30986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:51.770062 30986 hybrid_clock.cc:648] HybridClock initialized: now 1781888331769977 us; error 70 us; skew 500 ppm
I20260619 16:58:51.770509 30986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:51 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888331, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:51.774255 30986 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:58:51.776043 30986 webserver.cc:460] Webserver started at http://127.16.48.62:45707/ using document root <none> and password file <none>
I20260619 16:58:51.776947 30986 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:51.777076 30986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:51.783514 30986 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 16:58:51.786856 31002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:51.788553 30986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:58:51.788738 30986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "76b10fd3668f46ff82948228dfcd13c4"
format_stamp: "Formatted at 2026-06-19 16:58:48 on dist-test-slave-3ffd"
server_key: "753d8fc86897ff57db7be198b86403ec"
server_key_iv: "ded441724c4de97a59d060e428f53e58"
server_key_version: "encryptionkey@0"
I20260619 16:58:51.789904 30986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:51.804031 30986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:51.808874 30986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:51.809146 30986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:51.830278 30986 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:42315
I20260619 16:58:51.830296 31053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:42315 every 8 connection(s)
I20260619 16:58:51.832157 30986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:58:51.838100 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 30986
I20260619 16:58:51.838958 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:42463
--local_ip_for_outbound_sockets=127.16.48.1
--tserver_master_addrs=127.16.48.62:42315
--webserver_port=34435
--webserver_interface=127.16.48.1
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:58:51.840266 31054 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:58:51.843794 31054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Bootstrap starting.
I20260619 16:58:51.864609 31054 log.cc:824] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:51.877125 31054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:58:51.877699 31054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Bootstrap complete.
I20260619 16:58:51.887316 31054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:51.888340 31054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b10fd3668f46ff82948228dfcd13c4, State: Initialized, Role: FOLLOWER
I20260619 16:58:51.889015 31054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:51.889282 31054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:58:51.889415 31054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:58:51.889593 31054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:58:51.893814 31054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:51.894451 31054 leader_election.cc:302] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b10fd3668f46ff82948228dfcd13c4; no voters:
I20260619 16:58:51.895149 31054 leader_election.cc:288] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:58:51.895324 31058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:58:51.896453 31058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 2 LEADER]: Becoming Leader. State: Replica: 76b10fd3668f46ff82948228dfcd13c4, State: Running, Role: LEADER
I20260619 16:58:51.897083 31058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:51.897384 31054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:58:51.899596 31059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "76b10fd3668f46ff82948228dfcd13c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } } }
I20260619 16:58:51.899971 31059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:51.905309 31060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b10fd3668f46ff82948228dfcd13c4. Latest consensus state: current_term: 2 leader_uuid: "76b10fd3668f46ff82948228dfcd13c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } } }
I20260619 16:58:51.905773 31060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:51.907356 31063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:58:51.916314 31063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:58:51.918042 31063 catalog_manager.cc:1259] Loaded cluster ID: 9f6685d7f46349cdbae402ec4a937484
I20260619 16:58:51.918227 31063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:58:51.921761 31063 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 16:58:51.923098 31075 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:58:51.939599 31063 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 16:58:51.949489 31063 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:58:51.951308 31063 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Loaded TSK: 0
I20260619 16:58:51.952973 31063 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:58:52.212579 31056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:52.212932 31056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:52.213091 31056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:52.220849 31056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:52.221014 31056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:52.221220 31056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:58:52.231670 31056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:42463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=34435
--tserver_master_addrs=127.16.48.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:52.233108 31056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:52.234472 31056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:52.250572 31081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:52.250598 31082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:52.251286 31084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:52.251288 31056 server_base.cc:1034] running on GCE node
I20260619 16:58:52.252331 31056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:52.253598 31056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:52.254851 31056 hybrid_clock.cc:648] HybridClock initialized: now 1781888332254795 us; error 47 us; skew 500 ppm
I20260619 16:58:52.255275 31056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:52 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888332, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:52.265244 31056 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:58:52.267318 31056 webserver.cc:460] Webserver started at http://127.16.48.1:34435/ using document root <none> and password file <none>
I20260619 16:58:52.268361 31056 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:52.268543 31056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:52.276374 31056 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:58:52.280077 31091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:52.282043 31056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:58:52.282239 31056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "a99cb23811a041abb005505b6d958daf"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "b64d4274d6f4e4f9b88a73c18d5bb55c"
server_key_iv: "d87eb4298e3b6b7c6dd1096cedd9f2d1"
server_key_version: "encryptionkey@0"
I20260619 16:58:52.283746 31056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:52.312424 31056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:52.317226 31056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:52.317480 31056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:52.318538 31056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:52.320030 31056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:52.320204 31056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:52.320343 31056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:52.320437 31056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:52.355023 31056 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:42463
I20260619 16:58:52.355084 31203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:42463 every 8 connection(s)
I20260619 16:58:52.356774 31056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:58:52.364478 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 31056
I20260619 16:58:52.365325 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:42527
--local_ip_for_outbound_sockets=127.16.48.2
--tserver_master_addrs=127.16.48.62:42315
--webserver_port=41073
--webserver_interface=127.16.48.2
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:58:52 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888332, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:52.385638 31207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:52.360543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:52285)
0619 16:58:52.361349 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:52.361395 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:52.363183 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:52.364984 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:52.364993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:52.365680 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:52.366367 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:52.366397 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:52.368992 (+ 2595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:52.368997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:52.370963 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:52.370979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:52.372596 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:52.372601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:52.372801 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:52.373548 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:52.373579 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:52.376256 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:52.380095 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:52.380110 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:52.380187 (+ 77us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:52.380719 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:52.381419 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:52.381425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:52.381429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:52.381739 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:52.382716 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:52.382725 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:52.383140 (+ 415us) client_negotiation.cc:769] Sending connection context
0619 16:58:52.383436 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 16:58:52.384099 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":319,"threads_started":1}
I20260619 16:58:52.385639 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:52.360216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:52285 (local address 127.16.48.62:42315)
0619 16:58:52.360823 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:52.360835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:52.361537 (+ 702us) server_negotiation.cc:407] Connection header received
0619 16:58:52.363774 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:52.363810 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:52.364391 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:52.364849 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:52.366664 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:52.368771 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:52.371144 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:52.372343 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:52.372574 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:52.376617 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:52.376627 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:52.376645 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:52.376694 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:52.379848 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:52.380900 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:52.380908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:52.380915 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:52.381236 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:52.381923 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:52.381930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:52.381936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:52.382544 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:52.382765 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:52.383693 (+ 928us) server_negotiation.cc:299] Negotiation successful
0619 16:58:52.384172 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":238,"threads_started":1}
I20260619 16:58:52.388803 31204 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42315
I20260619 16:58:52.389256 31204 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:52.390460 31204 heartbeater.cc:511] Master 127.16.48.62:42315 requested a full tablet report, sending...
I20260619 16:58:52.393096 31019 ts_manager.cc:194] Registered new tserver with Master: a99cb23811a041abb005505b6d958daf (127.16.48.1:42463)
I20260619 16:58:52.394906 31019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:52285
W20260619 16:58:52.729749 31208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:52.730074 31208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:52.730225 31208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:52.738022 31208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:52.738157 31208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:52.738348 31208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:58:52.748669 31208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:42527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=41073
--tserver_master_addrs=127.16.48.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:52.750062 31208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:52.751590 31208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:52.765730 31214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:52.765821 31216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:52.765753 31213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:52.766923 31208 server_base.cc:1034] running on GCE node
I20260619 16:58:52.767611 31208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:52.768751 31208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:52.769991 31208 hybrid_clock.cc:648] HybridClock initialized: now 1781888332769926 us; error 62 us; skew 500 ppm
I20260619 16:58:52.770355 31208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:52 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888332, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:52.773783 31208 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:58:52.775463 31208 webserver.cc:460] Webserver started at http://127.16.48.2:41073/ using document root <none> and password file <none>
I20260619 16:58:52.776300 31208 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:52.776468 31208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:52.782698 31208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:58:52.785787 31223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:52.787424 31208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:58:52.787655 31208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "fc55e6575ee148708d592d15a00d54ca"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "c4b987cea9fecad0360e16ec74753f06"
server_key_iv: "db4465676c87649b0ff8f9fba7736661"
server_key_version: "encryptionkey@0"
I20260619 16:58:52.788833 31208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:52.824288 31208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:52.829087 31208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:52.829411 31208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:52.830895 31208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:52.832878 31208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:52.832998 31208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:52.833106 31208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:52.833176 31208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:52.869155 31208 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:42527
I20260619 16:58:52.869239 31335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:42527 every 8 connection(s)
I20260619 16:58:52.870762 31208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:58:52.874233 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 31208
I20260619 16:58:52.875043 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:45753
--local_ip_for_outbound_sockets=127.16.48.3
--tserver_master_addrs=127.16.48.62:42315
--webserver_port=45849
--webserver_interface=127.16.48.3
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:58:52 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888332, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:52.894449 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:52.873635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:33605 (local address 127.16.48.62:42315)
0619 16:58:52.873841 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:52.873846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:52.875562 (+ 1716us) server_negotiation.cc:407] Connection header received
0619 16:58:52.877883 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:52.877889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:52.878008 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:52.878136 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:52.879850 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:52.880655 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:52.882364 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:52.882793 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:52.882934 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:52.887645 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:52.887652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:52.887655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:52.887696 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:52.890575 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:52.891513 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:52.891526 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:52.891529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:52.891691 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:52.892234 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:52.892244 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:52.892246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:52.892544 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:52.892723 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:52.893732 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0619 16:58:52.894096 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 16:58:52.895546 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:52.874324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:33605)
0619 16:58:52.875339 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:52.875407 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:52.877495 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:52.878412 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:52.878421 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:52.879003 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:52.879641 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:52.879660 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:52.880841 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:52.880846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:52.882175 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:52.882188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:52.883056 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:52.883064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:52.883440 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:52.884302 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:52.884338 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:52.887330 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:52.890803 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:52.890815 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:52.890842 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:52.891359 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:52.891820 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:52.891825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:52.891829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:52.892052 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:52.892693 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:52.892703 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:52.893212 (+ 509us) client_negotiation.cc:769] Sending connection context
0619 16:58:52.893562 (+ 350us) client_negotiation.cc:240] Negotiation successful
0619 16:58:52.894196 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":403,"threads_started":1}
I20260619 16:58:52.897272 31336 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42315
I20260619 16:58:52.897691 31336 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:52.898794 31336 heartbeater.cc:511] Master 127.16.48.62:42315 requested a full tablet report, sending...
I20260619 16:58:52.900799 31019 ts_manager.cc:194] Registered new tserver with Master: fc55e6575ee148708d592d15a00d54ca (127.16.48.2:42527)
I20260619 16:58:52.901729 31019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:33605
W20260619 16:58:53.234541 31339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:53.234841 31339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:53.235001 31339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:53.242764 31339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:53.242900 31339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:53.243095 31339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:58:53.254040 31339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:45753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=45849
--tserver_master_addrs=127.16.48.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:53.255493 31339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:53.256995 31339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:53.271407 31347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:53.271449 31345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:53.271373 31344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:53.272291 31339 server_base.cc:1034] running on GCE node
I20260619 16:58:53.273084 31339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:53.274227 31339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:53.275480 31339 hybrid_clock.cc:648] HybridClock initialized: now 1781888333275414 us; error 67 us; skew 500 ppm
I20260619 16:58:53.275843 31339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:53 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888333, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:53.279395 31339 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:58:53.281241 31339 webserver.cc:460] Webserver started at http://127.16.48.3:45849/ using document root <none> and password file <none>
I20260619 16:58:53.281997 31339 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:53.282167 31339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:53.288008 31339 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:58:53.291312 31354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:53.293022 31339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:58:53.293241 31339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "571c05a4dbfb496ab8fb07213beda5d4"
format_stamp: "Formatted at 2026-06-19 16:58:50 on dist-test-slave-3ffd"
server_key: "09f9c51c88e368d4d528597078aafe72"
server_key_iv: "d38faa49c30f8e9d911bf89b169df19a"
server_key_version: "encryptionkey@0"
I20260619 16:58:53.294847 31339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:53.310621 31339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:53.315254 31339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:53.315569 31339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:53.316758 31339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:53.318287 31339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:53.318429 31339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:53.318562 31339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:53.318652 31339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:53.354143 31339 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:45753
I20260619 16:58:53.354203 31466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:45753 every 8 connection(s)
I20260619 16:58:53.355751 31339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:58:53.362798 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 31339
Jun 19 16:58:53 dist-test-slave-3ffd krb5kdc[30434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888333, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:58:53.381448 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.363864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.16.48.62:42315)
0619 16:58:53.364444 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.364450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.364927 (+ 477us) server_negotiation.cc:407] Connection header received
0619 16:58:53.365077 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.365083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.365177 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.365301 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:53.367285 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.368476 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.370421 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.371081 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.371224 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.374339 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:53.374344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:53.374347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:53.374407 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:53.378371 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:53.379200 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:53.379206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:53.379210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:53.379362 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:53.379766 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:53.379772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:53.379775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:53.380094 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:53.380397 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.380745 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.381007 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"spinlock_wait_cycles":768,"thread_start_us":344,"threads_started":1}
I20260619 16:58:53.381822 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.358991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:39899 (local address 127.16.48.62:42315)
0619 16:58:53.359207 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.359214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.360650 (+ 1436us) server_negotiation.cc:407] Connection header received
0619 16:58:53.363039 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.363050 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.363129 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.363227 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:53.365502 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.366743 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.369131 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.369760 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.369912 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.374307 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:53.374313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:53.374316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:53.374357 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:53.378415 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:53.379340 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:53.379345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:53.379348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:53.379465 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:53.379964 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:53.379969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:53.379972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:53.380298 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:53.380453 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.381227 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.381442 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"spinlock_wait_cycles":2944}
I20260619 16:58:53.383106 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.359512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:39899)
0619 16:58:53.360439 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:53.360491 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:53.362554 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:53.363558 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:53.363568 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:53.364401 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:58:53.365282 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.365314 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.366925 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.366929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.368963 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.368976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.370029 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.370035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.370258 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.371113 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:58:53.371153 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:58:53.374011 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:58:53.378633 (+ 4622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:53.378645 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:53.378668 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:53.379184 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:53.379593 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:58:53.379598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:58:53.379602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:58:53.379827 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:58:53.380431 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:58:53.380440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:58:53.380822 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 16:58:53.381123 (+ 301us) client_negotiation.cc:240] Negotiation successful
0619 16:58:53.381722 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":337,"threads_started":1}
I20260619 16:58:53.384624 31467 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42315
I20260619 16:58:53.384912 31467 heartbeater.cc:464] Registering TS with master...
I20260619 16:58:53.385670 31467 heartbeater.cc:511] Master 127.16.48.62:42315 requested a full tablet report, sending...
I20260619 16:58:53.387236 31018 ts_manager.cc:194] Registered new tserver with Master: 571c05a4dbfb496ab8fb07213beda5d4 (127.16.48.3:45753)
I20260619 16:58:53.388267 31018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:39899
I20260619 16:58:53.397086 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:58:53.399202 31204 heartbeater.cc:503] Master 127.16.48.62:42315 was elected leader, sending a full tablet report...
I20260619 16:58:53.404630 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.398094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.16.48.62:42315)
0619 16:58:53.398352 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.398362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.398723 (+ 361us) server_negotiation.cc:407] Connection header received
0619 16:58:53.398958 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.398963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.399136 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.399351 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:58:53.399826 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.400894 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.402451 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.402883 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.402988 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.403455 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:58:53.404004 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:58:53.404204 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.404270 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.404344 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 16:58:53.405678 31018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:58:53.413437 31018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:58:53.460318 31477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.426185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:42527 (local address 127.0.0.1:48840)
0619 16:58:53.442822 (+ 16637us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:53.442898 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:53.443189 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:53.451724 (+ 8535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:53.451733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:53.451766 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:53.452493 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.452510 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.454531 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.454537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.457565 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.457576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.459091 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.459098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.459297 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.459370 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 16:58:53.459519 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 16:58:53.459756 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16242,"mutex_wait_us":619,"spinlock_wait_cycles":13056,"thread_start_us":16052,"threads_started":1}
I20260619 16:58:53.461006 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.428931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.16.48.1:42463)
0619 16:58:53.449339 (+ 20408us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.449355 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.452086 (+ 2731us) server_negotiation.cc:407] Connection header received
0619 16:58:53.452270 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.452284 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.452550 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.452761 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:53.453331 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.454814 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.457472 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.458957 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.459095 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.459336 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.460468 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.460651 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20109,"thread_start_us":20425,"threads_started":1}
I20260619 16:58:53.461051 31481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.426324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:45753 (local address 127.0.0.1:57714)
0619 16:58:53.451030 (+ 24706us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:53.451059 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:53.451240 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:53.453598 (+ 2358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:53.453603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:53.453630 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:53.454002 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.454014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.456145 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.456150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.457954 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.457968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.460147 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.460158 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.460340 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.460385 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 16:58:53.460480 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:58:53.460621 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24497,"mutex_wait_us":2,"thread_start_us":25226,"threads_started":1}
I20260619 16:58:53.461333 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.426344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.16.48.2:42527)
0619 16:58:53.450164 (+ 23820us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.450177 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.450222 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:58:53.450372 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.450385 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.451400 (+ 1015us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.451714 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:53.452893 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.454365 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.457750 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.458866 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.459040 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.459280 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.459629 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.459826 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23578,"thread_start_us":23468,"threads_started":1}
I20260619 16:58:53.461433 31480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.426716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.16.48.3:45753)
0619 16:58:53.453004 (+ 26288us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.453013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.453048 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:58:53.453141 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.453147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.453379 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.453572 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:53.454215 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.455717 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.458136 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.459574 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.459721 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.459940 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.460911 (+ 971us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.461120 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25749,"thread_start_us":25577,"threads_started":1}
I20260619 16:58:53.460355 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.428694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.0.0.1:48548)
0619 16:58:53.451799 (+ 23105us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:53.451825 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:53.451970 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:53.452801 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:53.452805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:53.452837 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:53.453151 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.453161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.454979 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.454984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.457306 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.457317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.459462 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.459471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.459662 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.459692 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:58:53.459775 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:58:53.459903 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22984,"mutex_wait_us":93,"spinlock_wait_cycles":7296,"thread_start_us":23343,"threads_started":1}
I20260619 16:58:53.465445 31402 tablet_service.cc:1467] Processing CreateTablet for tablet d92e5af9600348b4a9b50641e9673e8c (DEFAULT_TABLE table=test-table [id=6535bebd4c314fd3add1bd9849ead7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:58:53.465447 31139 tablet_service.cc:1467] Processing CreateTablet for tablet d92e5af9600348b4a9b50641e9673e8c (DEFAULT_TABLE table=test-table [id=6535bebd4c314fd3add1bd9849ead7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:58:53.465405 31271 tablet_service.cc:1467] Processing CreateTablet for tablet d92e5af9600348b4a9b50641e9673e8c (DEFAULT_TABLE table=test-table [id=6535bebd4c314fd3add1bd9849ead7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:58:53.467053 31402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d92e5af9600348b4a9b50641e9673e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:53.467073 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d92e5af9600348b4a9b50641e9673e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:53.467053 31271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d92e5af9600348b4a9b50641e9673e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:58:53.482398 31483 tablet_bootstrap.cc:492] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: Bootstrap starting.
I20260619 16:58:53.483959 31484 tablet_bootstrap.cc:492] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Bootstrap starting.
I20260619 16:58:53.484935 31485 tablet_bootstrap.cc:492] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: Bootstrap starting.
I20260619 16:58:53.487834 31484 tablet_bootstrap.cc:654] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:53.487916 31483 tablet_bootstrap.cc:654] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:53.488786 31485 tablet_bootstrap.cc:654] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: Neither blocks nor log segments found. Creating new log.
I20260619 16:58:53.489188 31484 log.cc:824] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:53.489717 31483 log.cc:824] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:53.490106 31485 log.cc:824] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:53.497457 31485 tablet_bootstrap.cc:492] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: No bootstrap required, opened a new log
I20260619 16:58:53.497457 31483 tablet_bootstrap.cc:492] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: No bootstrap required, opened a new log
I20260619 16:58:53.497797 31483 ts_tablet_manager.cc:1397] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260619 16:58:53.497797 31485 ts_tablet_manager.cc:1397] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260619 16:58:53.498265 31484 tablet_bootstrap.cc:492] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: No bootstrap required, opened a new log
I20260619 16:58:53.498605 31484 ts_tablet_manager.cc:1397] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260619 16:58:53.506700 31484 raft_consensus.cc:348] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } }
I20260619 16:58:53.507186 31484 raft_consensus.cc:374] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:53.507339 31484 raft_consensus.cc:729] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99cb23811a041abb005505b6d958daf, State: Initialized, Role: FOLLOWER
I20260619 16:58:53.508059 31484 consensus_queue.cc:260] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } }
I20260619 16:58:53.509896 31484 ts_tablet_manager.cc:1428] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260619 16:58:53.509596 31483 raft_consensus.cc:348] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } }
I20260619 16:58:53.510066 31483 raft_consensus.cc:374] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:53.510179 31483 raft_consensus.cc:729] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc55e6575ee148708d592d15a00d54ca, State: Initialized, Role: FOLLOWER
I20260619 16:58:53.510838 31483 consensus_queue.cc:260] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } }
I20260619 16:58:53.510720 31485 raft_consensus.cc:348] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } }
I20260619 16:58:53.511195 31485 raft_consensus.cc:374] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:58:53.511307 31485 raft_consensus.cc:729] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571c05a4dbfb496ab8fb07213beda5d4, State: Initialized, Role: FOLLOWER
I20260619 16:58:53.512025 31336 heartbeater.cc:503] Master 127.16.48.62:42315 was elected leader, sending a full tablet report...
I20260619 16:58:53.512492 31485 consensus_queue.cc:260] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } }
I20260619 16:58:53.514065 31467 heartbeater.cc:503] Master 127.16.48.62:42315 was elected leader, sending a full tablet report...
I20260619 16:58:53.515271 31485 ts_tablet_manager.cc:1428] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260619 16:58:53.516968 31483 ts_tablet_manager.cc:1428] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: Time spent starting tablet: real 0.019s user 0.012s sys 0.004s
W20260619 16:58:53.607649 31468 tablet.cc:2369] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:58:53.609251 31205 tablet.cc:2369] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:58:53.622987 31337 tablet.cc:2369] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:58:53.642688 31489 raft_consensus.cc:482] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:58:53.643107 31489 raft_consensus.cc:504] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } }
I20260619 16:58:53.645412 31489 leader_election.cc:288] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc55e6575ee148708d592d15a00d54ca (127.16.48.2:42527), 571c05a4dbfb496ab8fb07213beda5d4 (127.16.48.3:45753)
I20260619 16:58:53.652630 31480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.645945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:33603 (local address 127.16.48.3:45753)
0619 16:58:53.646171 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.646178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.647132 (+ 954us) server_negotiation.cc:407] Connection header received
0619 16:58:53.647377 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.647381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.647444 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.647573 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:53.648183 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.648916 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.650747 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.651585 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.651748 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.651870 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.652201 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.652342 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:58:53.652760 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.645927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:45753 (local address 127.16.48.1:33603)
0619 16:58:53.647006 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:53.647023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:53.647217 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:53.647605 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:53.647609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:53.647709 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:53.648013 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.648022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.649052 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.649058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.650573 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.650587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.651814 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.651820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.651991 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.652018 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:58:53.652128 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:58:53.652311 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"spinlock_wait_cycles":27520,"thread_start_us":1241,"threads_started":1}
I20260619 16:58:53.653358 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.645940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:36363 (local address 127.16.48.2:42527)
0619 16:58:53.646172 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.646178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.647593 (+ 1415us) server_negotiation.cc:407] Connection header received
0619 16:58:53.647816 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.647823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.647886 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.648002 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:58:53.648645 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.649384 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.651263 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.652177 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.652366 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.652486 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.652891 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.653042 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:58:53.654177 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.646259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:42527 (local address 127.16.48.1:36363)
0619 16:58:53.647462 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:53.647479 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:58:53.647652 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:58:53.648019 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:58:53.648024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:58:53.648048 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:58:53.648467 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.648477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.649546 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.649552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.651063 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:58:53.651078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.652451 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.652462 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:58:53.652639 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.652665 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:58:53.653603 (+ 938us) client_negotiation.cc:240] Negotiation successful
0619 16:58:53.653789 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":414,"threads_started":1}
I20260619 16:58:53.654263 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e5af9600348b4a9b50641e9673e8c" candidate_uuid: "a99cb23811a041abb005505b6d958daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571c05a4dbfb496ab8fb07213beda5d4" is_pre_election: true
I20260619 16:58:53.654942 31422 raft_consensus.cc:2454] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99cb23811a041abb005505b6d958daf in term 0.
I20260619 16:58:53.655297 31291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e5af9600348b4a9b50641e9673e8c" candidate_uuid: "a99cb23811a041abb005505b6d958daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc55e6575ee148708d592d15a00d54ca" is_pre_election: true
I20260619 16:58:53.655735 31093 leader_election.cc:302] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571c05a4dbfb496ab8fb07213beda5d4, a99cb23811a041abb005505b6d958daf; no voters:
I20260619 16:58:53.655870 31291 raft_consensus.cc:2454] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99cb23811a041abb005505b6d958daf in term 0.
I20260619 16:58:53.656213 31489 raft_consensus.cc:2781] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:58:53.656389 31489 raft_consensus.cc:482] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:58:53.656535 31489 raft_consensus.cc:3037] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:53.658989 31489 raft_consensus.cc:504] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } }
I20260619 16:58:53.659773 31489 leader_election.cc:288] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [CANDIDATE]: Term 1 election: Requested vote from peers fc55e6575ee148708d592d15a00d54ca (127.16.48.2:42527), 571c05a4dbfb496ab8fb07213beda5d4 (127.16.48.3:45753)
I20260619 16:58:53.660347 31291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e5af9600348b4a9b50641e9673e8c" candidate_uuid: "a99cb23811a041abb005505b6d958daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc55e6575ee148708d592d15a00d54ca"
I20260619 16:58:53.660393 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e5af9600348b4a9b50641e9673e8c" candidate_uuid: "a99cb23811a041abb005505b6d958daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571c05a4dbfb496ab8fb07213beda5d4"
I20260619 16:58:53.660598 31291 raft_consensus.cc:3037] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:53.660632 31422 raft_consensus.cc:3037] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:58:53.664027 31422 raft_consensus.cc:2454] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99cb23811a041abb005505b6d958daf in term 1.
I20260619 16:58:53.664021 31291 raft_consensus.cc:2454] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99cb23811a041abb005505b6d958daf in term 1.
I20260619 16:58:53.664646 31092 leader_election.cc:302] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a99cb23811a041abb005505b6d958daf, fc55e6575ee148708d592d15a00d54ca; no voters:
I20260619 16:58:53.665052 31489 raft_consensus.cc:2781] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:58:53.665589 31489 raft_consensus.cc:686] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 1 LEADER]: Becoming Leader. State: Replica: a99cb23811a041abb005505b6d958daf, State: Running, Role: LEADER
I20260619 16:58:53.666095 31489 consensus_queue.cc:237] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } }
I20260619 16:58:53.671058 31019 catalog_manager.cc:5495] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf reported cstate change: term changed from 0 to 1, leader changed from <none> to a99cb23811a041abb005505b6d958daf (127.16.48.1). New cstate: current_term: 1 leader_uuid: "a99cb23811a041abb005505b6d958daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a99cb23811a041abb005505b6d958daf" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42463 } health_report { overall_health: HEALTHY } } }
I20260619 16:58:53.706554 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.700845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.16.48.1:42463)
0619 16:58:53.701044 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.701048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.701067 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:58:53.701325 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.701330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.701419 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.701563 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:58:53.701936 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.702708 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.704366 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.704806 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.704924 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.705474 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:58:53.706025 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:58:53.706164 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.706245 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.706343 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:58:53.716200 31422 raft_consensus.cc:1261] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [term 1 FOLLOWER]: Refusing update from remote peer a99cb23811a041abb005505b6d958daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:58:53.716192 31291 raft_consensus.cc:1261] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [term 1 FOLLOWER]: Refusing update from remote peer a99cb23811a041abb005505b6d958daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:58:53.717414 31489 consensus_queue.cc:1035] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "571c05a4dbfb496ab8fb07213beda5d4" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:58:53.717762 31494 consensus_queue.cc:1035] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc55e6575ee148708d592d15a00d54ca" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:58:53.733387 31498 mvcc.cc:205] Tried to move back new op lower bound from 7298614614890246144 to 7298614614700052480. Current Snapshot: MvccSnapshot[applied={T|T < 7298614614890246144}]
I20260619 16:58:53.746416 31019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:58:53.746726 31019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:58:53.750437 31019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Sending DeleteTablet for 3 replicas of tablet d92e5af9600348b4a9b50641e9673e8c
I20260619 16:58:53.751816 31271 tablet_service.cc:1514] Processing DeleteTablet for tablet d92e5af9600348b4a9b50641e9673e8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:58:53 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:48840
I20260619 16:58:53.751821 31139 tablet_service.cc:1514] Processing DeleteTablet for tablet d92e5af9600348b4a9b50641e9673e8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:58:53 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:48548
I20260619 16:58:53.751842 31402 tablet_service.cc:1514] Processing DeleteTablet for tablet d92e5af9600348b4a9b50641e9673e8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:58:53 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:57714
I20260619 16:58:53.753739 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 31056
I20260619 16:58:53.766436 31507 tablet_replica.cc:330] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: stopping tablet replica
I20260619 16:58:53.767117 31507 raft_consensus.cc:2229] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:58:53.768247 31507 raft_consensus.cc:2258] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:58:53.769150 31509 tablet_replica.cc:330] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: stopping tablet replica
I20260619 16:58:53.769641 31509 raft_consensus.cc:2229] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:58:53.770215 31509 raft_consensus.cc:2258] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:58:53.771703 31508 tablet_replica.cc:330] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: stopping tablet replica
I20260619 16:58:53.772193 31508 raft_consensus.cc:2229] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:58:53.772763 31508 raft_consensus.cc:2258] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:58:53.774109 31507 ts_tablet_manager.cc:1905] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:58:53.774646 31509 ts_tablet_manager.cc:1905] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:58:53.777096 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.754842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.16.48.1:42463)
0619 16:58:53.755078 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.755086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.755126 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:58:53.755209 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.755217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.755323 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.755464 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:53.757932 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.759622 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.762099 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.763173 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.763358 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.768451 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:53.768465 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:53.768484 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:53.768535 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:53.773109 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:53.774224 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:53.774234 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:53.774237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:53.774417 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:53.774895 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:53.774900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:53.774903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:53.775416 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:53.775759 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.776391 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.776678 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:58:53.778116 31199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:53.780333 31508 ts_tablet_manager.cc:1905] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:58:53.785921 31509 ts_tablet_manager.cc:1918] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:58:53.786154 31509 log.cc:1196] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/d92e5af9600348b4a9b50641e9673e8c
I20260619 16:58:53.786630 31509 ts_tablet_manager.cc:1939] T d92e5af9600348b4a9b50641e9673e8c P fc55e6575ee148708d592d15a00d54ca: Deleting consensus metadata
I20260619 16:58:53.788398 31003 catalog_manager.cc:4841] TS fc55e6575ee148708d592d15a00d54ca (127.16.48.2:42527): tablet d92e5af9600348b4a9b50641e9673e8c (table test-table [id=6535bebd4c314fd3add1bd9849ead7a8]) successfully deleted
I20260619 16:58:53.788455 31508 ts_tablet_manager.cc:1918] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:58:53.788635 31508 log.cc:1196] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/d92e5af9600348b4a9b50641e9673e8c
I20260619 16:58:53.789225 31508 ts_tablet_manager.cc:1939] T d92e5af9600348b4a9b50641e9673e8c P 571c05a4dbfb496ab8fb07213beda5d4: Deleting consensus metadata
I20260619 16:58:53.791406 31004 catalog_manager.cc:4841] TS 571c05a4dbfb496ab8fb07213beda5d4 (127.16.48.3:45753): tablet d92e5af9600348b4a9b50641e9673e8c (table test-table [id=6535bebd4c314fd3add1bd9849ead7a8]) successfully deleted
I20260619 16:58:53.881913 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 31056
W20260619 16:58:53.902125 31006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.48.1:42463 (error 108)
I20260619 16:58:53.902259 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 31208
I20260619 16:58:53.903962 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.903431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.0.0.1:48562)
0619 16:58:53.903643 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:53.903773 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260619 16:58:53.904273 31006 catalog_manager.cc:4584] TS a99cb23811a041abb005505b6d958daf (127.16.48.1:42463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d92e5af9600348b4a9b50641e9673e8c: Network error: Client connection negotiation failed: client connection to 127.16.48.1:42463: connect: Connection refused (error 111)
I20260619 16:58:53.917068 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.903144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.16.48.2:42527)
0619 16:58:53.903369 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:53.903375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:53.903398 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:58:53.903558 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:53.903563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:53.903664 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:53.903747 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:53.905344 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.906117 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.907455 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:53.907938 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:53.908076 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:53.910828 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:53.910845 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:53.910863 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:53.910902 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:53.913897 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:53.914738 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:53.914747 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:53.914749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:53.914881 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:53.915317 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:53.915322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:53.915329 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:53.915756 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:53.916083 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:53.916441 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 16:58:53.916696 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 16:58:53.917819 31331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:53.958066 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.957441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.0.0.1:48564)
0619 16:58:53.957725 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:53.957859 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 16:58:53.959268 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:53.958850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.0.0.1:48572)
0619 16:58:53.959018 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:53.959122 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:58:54.009738 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 31208
I20260619 16:58:54.020172 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:54.019608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.0.0.1:48584)
0619 16:58:54.019839 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:54.019948 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 16:58:54.021332 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:54.020939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.0.0.1:48592)
0619 16:58:54.021103 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:54.021193 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:58:54.027415 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 31339
I20260619 16:58:54.043722 31480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:54.028391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.16.48.3:45753)
0619 16:58:54.028572 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:58:54.028577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:58:54.028596 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:58:54.028909 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:58:54.028916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:58:54.029001 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:58:54.029097 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:58:54.030696 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:54.031565 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:54.033369 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:58:54.033775 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:58:54.033890 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:58:54.037341 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:58:54.037358 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:58:54.037375 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:58:54.037422 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:58:54.040616 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:54.041699 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:54.041708 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:54.041710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:54.041866 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:58:54.042248 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:58:54.042251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:58:54.042253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:58:54.042621 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:58:54.042857 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 16:58:54.043126 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 16:58:54.043304 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:58:54.044708 31462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:54.101991 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:54.101470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.0.0.1:48602)
0619 16:58:54.101718 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:54.101806 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 16:58:54.103152 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:54.102716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42463 (local address 127.0.0.1:48604)
0619 16:58:54.102936 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:54.103029 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 16:58:54.135762 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 31339
I20260619 16:58:54.152825 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 30986
I20260619 16:58:54.154075 31049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:58:54.272398 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 30986
I20260619 16:58:54.295581 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42315
--webserver_interface=127.16.48.62
--webserver_port=45707
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:42315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:54.673291 31515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:54.673708 31515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:54.673879 31515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:54.681551 31515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:58:54.681737 31515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:54.681856 31515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:54.681959 31515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:58:54.682035 31515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:58:54.692492 31515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:42315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=45707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:54.694015 31515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:54.695645 31515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:54.709957 31521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:54.710000 31520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:54.710953 31523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:54.711746 31515 server_base.cc:1034] running on GCE node
I20260619 16:58:54.712687 31515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:54.714296 31515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:54.715593 31515 hybrid_clock.cc:648] HybridClock initialized: now 1781888334715537 us; error 53 us; skew 500 ppm
I20260619 16:58:54.716039 31515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:54 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888334, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:54.719532 31515 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:58:54.721246 31515 webserver.cc:460] Webserver started at http://127.16.48.62:45707/ using document root <none> and password file <none>
I20260619 16:58:54.722029 31515 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:54.722188 31515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:54.728253 31515 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 16:58:54.731679 31530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:54.733282 31515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:58:54.733494 31515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "76b10fd3668f46ff82948228dfcd13c4"
format_stamp: "Formatted at 2026-06-19 16:58:48 on dist-test-slave-3ffd"
server_key: "753d8fc86897ff57db7be198b86403ec"
server_key_iv: "ded441724c4de97a59d060e428f53e58"
server_key_version: "encryptionkey@0"
I20260619 16:58:54.734544 31515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:54.754442 31515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:54.759240 31515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:54.759649 31515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:54.780292 31515 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:42315
I20260619 16:58:54.780341 31581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:42315 every 8 connection(s)
I20260619 16:58:54.782008 31515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:58:54.789436 31582 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:58:54.790658 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 31515
I20260619 16:58:54.791728 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:42463
--local_ip_for_outbound_sockets=127.16.48.1
--tserver_master_addrs=127.16.48.62:42315
--webserver_port=34435
--webserver_interface=127.16.48.1
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:58:54.794829 31582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Bootstrap starting.
I20260619 16:58:54.818085 31582 log.cc:824] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:58:54.835866 31582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:58:54.836503 31582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4: Bootstrap complete.
I20260619 16:58:54.846500 31582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:54.847469 31582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b10fd3668f46ff82948228dfcd13c4, State: Initialized, Role: FOLLOWER
I20260619 16:58:54.848269 31582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:54.848534 31582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:58:54.848670 31582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:58:54.848872 31582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:58:54.853349 31582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:54.853966 31582 leader_election.cc:302] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b10fd3668f46ff82948228dfcd13c4; no voters:
I20260619 16:58:54.854686 31582 leader_election.cc:288] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:58:54.854830 31586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:58:54.855970 31586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [term 3 LEADER]: Becoming Leader. State: Replica: 76b10fd3668f46ff82948228dfcd13c4, State: Running, Role: LEADER
I20260619 16:58:54.856705 31586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } }
I20260619 16:58:54.857178 31582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:58:54.859488 31588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b10fd3668f46ff82948228dfcd13c4. Latest consensus state: current_term: 3 leader_uuid: "76b10fd3668f46ff82948228dfcd13c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } } }
I20260619 16:58:54.859968 31588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:54.860415 31587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "76b10fd3668f46ff82948228dfcd13c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b10fd3668f46ff82948228dfcd13c4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42315 } } }
I20260619 16:58:54.860714 31587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b10fd3668f46ff82948228dfcd13c4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:58:54.871614 31591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:58:54.881129 31591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:58:54.882961 31591 catalog_manager.cc:1259] Loaded cluster ID: 9f6685d7f46349cdbae402ec4a937484
I20260619 16:58:54.883179 31591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 16:58:54.885557 31591 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb19345d9ad google::LogMessage::Fail() at ??:0
@ 0x7fb19346088a google::LogMessage::SendToLog() at ??:0
@ 0x7fb19345d4de google::LogMessage::Flush() at ??:0
@ 0x7fb19345ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb1a8e90512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb1a8e8d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb1a8f15610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb1a8f15439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb1960a9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb1960ac91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb1960ac799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb196087eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb1993616db start_thread at ??:0
@ 0x7fb1922f761f clone at ??:0
W20260619 16:58:55.194161 31584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:55.194558 31584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:55.194753 31584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:55.202945 31584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:55.203145 31584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:55.203351 31584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:58:55.214079 31584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:42463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=34435
--tserver_master_addrs=127.16.48.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:55.215703 31584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:55.217258 31584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:55.232695 31624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:55.232642 31626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:55.232743 31623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:55.238159 31584 server_base.cc:1034] running on GCE node
I20260619 16:58:55.238968 31584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:55.240293 31584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:55.241566 31584 hybrid_clock.cc:648] HybridClock initialized: now 1781888335241511 us; error 55 us; skew 500 ppm
I20260619 16:58:55.241962 31584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:55 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888335, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:55.245600 31584 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:58:55.247455 31584 webserver.cc:460] Webserver started at http://127.16.48.1:34435/ using document root <none> and password file <none>
I20260619 16:58:55.248420 31584 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:55.248592 31584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:55.254885 31584 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:58:55.258508 31633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:55.260586 31584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:58:55.260810 31584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "a99cb23811a041abb005505b6d958daf"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "b64d4274d6f4e4f9b88a73c18d5bb55c"
server_key_iv: "d87eb4298e3b6b7c6dd1096cedd9f2d1"
server_key_version: "encryptionkey@0"
I20260619 16:58:55.261935 31584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:55.282895 31584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:55.287595 31584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:55.287925 31584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:55.289124 31584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:55.291129 31640 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 16:58:55.296998 31640 ts_tablet_manager.cc:1852] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 16:58:55.298062 31640 ts_tablet_manager.cc:1905] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:58:55.306077 31640 ts_tablet_manager.cc:1918] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:58:55.306337 31640 log.cc:1196] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/d92e5af9600348b4a9b50641e9673e8c
I20260619 16:58:55.306881 31640 ts_tablet_manager.cc:1939] T d92e5af9600348b4a9b50641e9673e8c P a99cb23811a041abb005505b6d958daf: Deleting consensus metadata
I20260619 16:58:55.307348 31584 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 16:58:55.307511 31584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260619 16:58:55.307665 31584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:55.307760 31584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:55.344079 31584 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:42463
I20260619 16:58:55.344209 31746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:42463 every 8 connection(s)
I20260619 16:58:55.345595 31584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:58:55.351320 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:55.349135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:37667)
0619 16:58:55.350020 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:55.350290 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":360,"threads_started":1}
I20260619 16:58:55.351482 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 31584
W20260619 16:58:55.352070 31747 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:42315 (0 consecutive failures): Network error: Failed to ping master at 127.16.48.62:42315: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
I20260619 16:58:55.352319 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:42527
--local_ip_for_outbound_sockets=127.16.48.2
--tserver_master_addrs=127.16.48.62:42315
--webserver_port=41073
--webserver_interface=127.16.48.2
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:58:55.353152 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:55.352765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:45855)
0619 16:58:55.352945 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:55.353034 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 16:58:55.354260 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:55.353752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:38985)
0619 16:58:55.353967 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:55.354078 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 16:58:55.354533 31747 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:58:55.706072 31750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:55.706471 31750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:55.706657 31750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:55.714272 31750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:55.714468 31750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:55.714672 31750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:58:55.725078 31750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:42527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=41073
--tserver_master_addrs=127.16.48.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:55.726598 31750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:55.728075 31750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:55.742758 31758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:55.742976 31755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:55.742970 31756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:55.744518 31750 server_base.cc:1034] running on GCE node
I20260619 16:58:55.745245 31750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:55.746423 31750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:55.747654 31750 hybrid_clock.cc:648] HybridClock initialized: now 1781888335747609 us; error 32 us; skew 500 ppm
I20260619 16:58:55.747980 31750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:55 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888335, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:55.751575 31750 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:58:55.753386 31750 webserver.cc:460] Webserver started at http://127.16.48.2:41073/ using document root <none> and password file <none>
I20260619 16:58:55.754148 31750 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:55.754271 31750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:55.760682 31750 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:58:55.763854 31765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:55.765573 31750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:58:55.765744 31750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "fc55e6575ee148708d592d15a00d54ca"
format_stamp: "Formatted at 2026-06-19 16:58:49 on dist-test-slave-3ffd"
server_key: "c4b987cea9fecad0360e16ec74753f06"
server_key_iv: "db4465676c87649b0ff8f9fba7736661"
server_key_version: "encryptionkey@0"
I20260619 16:58:55.766901 31750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:55.783047 31750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:55.788075 31750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:55.788419 31750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:55.789456 31750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:55.790923 31750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:55.791016 31750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:55.791105 31750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:55.791162 31750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:55.826671 31750 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:42527
I20260619 16:58:55.826727 31877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:42527 every 8 connection(s)
I20260619 16:58:55.828464 31750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:58:55.834152 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:55.831928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:56869)
0619 16:58:55.832867 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:55.833109 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":416,"threads_started":1}
W20260619 16:58:55.835083 31878 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:42315 (0 consecutive failures): Network error: Failed to ping master at 127.16.48.62:42315: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
I20260619 16:58:55.836200 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:55.835724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:46627)
0619 16:58:55.835912 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:55.836007 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 16:58:55.837088 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:55.836712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:57795)
0619 16:58:55.836872 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:55.836970 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 16:58:55.837385 31878 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:58:55.837819 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 31750
I20260619 16:58:55.838418 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:45753
--local_ip_for_outbound_sockets=127.16.48.3
--tserver_master_addrs=127.16.48.62:42315
--webserver_port=45849
--webserver_interface=127.16.48.3
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:58:56.187052 31881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:58:56.187439 31881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:58:56.187629 31881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:58:56.195262 31881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:58:56.195464 31881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:58:56.195665 31881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:58:56.206158 31881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:45753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=45849
--tserver_master_addrs=127.16.48.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:58:56.207587 31881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:58:56.209039 31881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:58:56.223814 31886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:56.223852 31889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:58:56.223759 31887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:58:56.225013 31881 server_base.cc:1034] running on GCE node
I20260619 16:58:56.225826 31881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:58:56.226909 31881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:58:56.228216 31881 hybrid_clock.cc:648] HybridClock initialized: now 1781888336228164 us; error 55 us; skew 500 ppm
I20260619 16:58:56.228581 31881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:58:56 dist-test-slave-3ffd krb5kdc[30434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:58:56.232370 31881 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:58:56.233959 31881 webserver.cc:460] Webserver started at http://127.16.48.3:45849/ using document root <none> and password file <none>
I20260619 16:58:56.234747 31881 fs_manager.cc:359] Metadata directory not provided
I20260619 16:58:56.234906 31881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:58:56.241005 31881 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:58:56.244099 31896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:56.245930 31881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:58:56.246145 31881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "571c05a4dbfb496ab8fb07213beda5d4"
format_stamp: "Formatted at 2026-06-19 16:58:50 on dist-test-slave-3ffd"
server_key: "09f9c51c88e368d4d528597078aafe72"
server_key_iv: "d38faa49c30f8e9d911bf89b169df19a"
server_key_version: "encryptionkey@0"
I20260619 16:58:56.247222 31881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:58:56.270619 31881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:58:56.275281 31881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:58:56.275597 31881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:58:56.276718 31881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:58:56.278199 31881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:58:56.278337 31881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:56.278467 31881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:58:56.278558 31881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:58:56.314633 31881 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:45753
I20260619 16:58:56.314731 32008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:45753 every 8 connection(s)
I20260619 16:58:56.316278 31881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:58:56.321832 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:56.319583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:35929)
0619 16:58:56.320515 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:56.320758 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":336,"threads_started":1}
W20260619 16:58:56.322793 32009 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:42315 (0 consecutive failures): Network error: Failed to ping master at 127.16.48.62:42315: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
I20260619 16:58:56.323796 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:56.323380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:51561)
0619 16:58:56.323574 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:56.323665 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 16:58:56.324432 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 31881
I20260619 16:58:56.324977 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:56.324556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:40285)
0619 16:58:56.324728 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:56.324824 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 16:58:56.325309 32009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:58:56.356225 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:56.355208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:39835)
0619 16:58:56.355756 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:56.355869 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":271,"threads_started":1}
I20260619 16:58:56.838887 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:56.837985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:56161)
0619 16:58:56.838519 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:56.838635 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":249,"threads_started":1}
I20260619 16:58:57.326897 32015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:57.325969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:45899)
0619 16:58:57.326480 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:57.326589 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":252,"threads_started":1}
I20260619 16:58:57.358306 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:57.357386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:55805)
0619 16:58:57.358031 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:57.358130 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":344,"threads_started":1}
I20260619 16:58:57.840777 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:57.839925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:53545)
0619 16:58:57.840496 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:57.840589 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":318,"threads_started":1}
I20260619 16:58:58.328823 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:58.328007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:40251)
0619 16:58:58.328538 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:58.328651 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":279,"threads_started":1}
I20260619 16:58:58.360545 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:58.359458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:60103)
0619 16:58:58.360060 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:58.360277 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":294,"threads_started":1}
I20260619 16:58:58.842571 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:58.841815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:51645)
0619 16:58:58.842301 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:58.842393 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":263,"threads_started":1}
I20260619 16:58:59.330772 32021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:59.329923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:35479)
0619 16:58:59.330473 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:59.330569 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":324,"threads_started":1}
I20260619 16:58:59.362522 32022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:59.361742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:51401)
0619 16:58:59.362256 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:59.362350 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":248,"threads_started":1}
I20260619 16:58:59.844460 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:58:59.843631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:57015)
0619 16:58:59.844088 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 16:58:59.844249 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260619 16:59:00.332748 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:00.331895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:58321)
0619 16:59:00.332434 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:00.332530 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":302,"threads_started":1}
I20260619 16:59:00.364636 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:00.363693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:54523)
0619 16:59:00.364351 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:00.364449 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":355,"threads_started":1}
I20260619 16:59:00.846380 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:00.845517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:41247)
0619 16:59:00.846087 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:00.846198 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":301,"threads_started":1}
I20260619 16:59:01.334749 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:01.333876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:47287)
0619 16:59:01.334434 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:01.334531 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":250,"threads_started":1}
I20260619 16:59:01.366653 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:01.365859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:51115)
0619 16:59:01.366376 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:01.366471 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":285,"threads_started":1}
I20260619 16:59:01.848455 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:01.847505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:44991)
0619 16:59:01.848022 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:01.848271 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
I20260619 16:59:02.336762 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:02.335783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:43031)
0619 16:59:02.336414 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:02.336521 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":393,"threads_started":1}
I20260619 16:59:02.368706 32031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:02.367774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:34361)
0619 16:59:02.368363 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:02.368457 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":237,"threads_started":1}
I20260619 16:59:02.851112 32032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:02.850018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:40935)
0619 16:59:02.850680 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:02.850823 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":300,"threads_started":1}
I20260619 16:59:03.339161 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:03.338163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:42779)
0619 16:59:03.338792 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:03.338900 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":308,"threads_started":1}
I20260619 16:59:03.370632 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:03.369796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:33959)
0619 16:59:03.370296 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:03.370393 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":255,"threads_started":1}
I20260619 16:59:03.853111 32035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:03.852331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:46771)
0619 16:59:03.852857 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:03.852939 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":316,"threads_started":1}
I20260619 16:59:04.341446 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:04.340636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:55159)
0619 16:59:04.341157 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:04.341270 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":265,"threads_started":1}
I20260619 16:59:04.373124 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:04.371909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:46211)
0619 16:59:04.372817 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:04.372935 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":631,"threads_started":1}
I20260619 16:59:04.854977 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:04.854139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:58173)
0619 16:59:04.854692 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:04.854785 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":254,"threads_started":1}
I20260619 16:59:05.343485 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:05.342669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:44943)
0619 16:59:05.343203 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:05.343300 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":255,"threads_started":1}
I20260619 16:59:05.374969 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:05.374211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:40333)
0619 16:59:05.374703 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:05.374797 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
I20260619 16:59:05.856957 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:05.856188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:35017)
0619 16:59:05.856698 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:05.856787 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":264,"threads_started":1}
I20260619 16:59:06.345774 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:06.344869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:40303)
0619 16:59:06.345398 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:06.345509 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":279,"threads_started":1}
I20260619 16:59:06.376963 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:06.376085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:33661)
0619 16:59:06.376654 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:06.376759 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":255,"threads_started":1}
I20260619 16:59:06.858981 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:06.858094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:52031)
0619 16:59:06.858645 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:06.858739 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":298,"threads_started":1}
I20260619 16:59:07.347882 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:07.347056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:36079)
0619 16:59:07.347583 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:07.347699 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":265,"threads_started":1}
I20260619 16:59:07.378818 32046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:07.378067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:48315)
0619 16:59:07.378536 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:07.378632 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260619 16:59:07.860867 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:07.860042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:55391)
0619 16:59:07.860607 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:07.860700 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":318,"threads_started":1}
I20260619 16:59:08.349963 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:08.349053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:57791)
0619 16:59:08.349707 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:08.349803 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":376,"threads_started":1}
I20260619 16:59:08.380932 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:08.380016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:45477)
0619 16:59:08.380598 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:08.380738 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":329,"threads_started":1}
I20260619 16:59:08.863059 32050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:08.862084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:41429)
0619 16:59:08.862638 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:08.862815 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":251,"threads_started":1}
I20260619 16:59:09.352007 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:09.351113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:59693)
0619 16:59:09.351632 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:09.351741 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":266,"threads_started":1}
I20260619 16:59:09.382741 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:09.381962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:38775)
0619 16:59:09.382455 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:09.382545 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":262,"threads_started":1}
I20260619 16:59:09.865067 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:09.864271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:46707)
0619 16:59:09.864769 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:09.864862 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":245,"threads_started":1}
I20260619 16:59:10.353858 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:10.353078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:52935)
0619 16:59:10.353562 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:10.353659 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":244,"threads_started":1}
I20260619 16:59:10.384774 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:10.383953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.1:45913)
0619 16:59:10.384488 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:10.384585 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":311,"threads_started":1}
I20260619 16:59:10.867276 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:10.866397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.2:58199)
0619 16:59:10.866959 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:10.867084 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":319,"threads_started":1}
I20260619 16:59:11.327436 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 31584
I20260619 16:59:11.355417 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:11.328691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.16.48.1:42463)
0619 16:59:11.329404 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:11.329425 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:11.329483 (+ 58us) server_negotiation.cc:407] Connection header received
0619 16:59:11.331896 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:11.331908 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:11.332499 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:11.332838 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:11.334321 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:11.335595 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:11.337008 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:11.337756 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:11.337914 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:11.346836 (+ 8922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:11.346849 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:11.346875 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:11.346921 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:11.351135 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:11.352289 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:11.352299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:11.352307 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:11.352561 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:11.353097 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:11.353102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:11.353105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:11.353756 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:11.354032 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:11.354662 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 16:59:11.355113 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":350,"threads_started":1}
I20260619 16:59:11.355988 32058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:11.355120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42315 (local address 127.16.48.3:37111)
0619 16:59:11.355624 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:11.355769 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":281,"threads_started":1}
I20260619 16:59:11.357177 31742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:11.445266 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 31584
I20260619 16:59:11.464643 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 31750
I20260619 16:59:11.484381 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:11.465808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.16.48.2:42527)
0619 16:59:11.466299 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:11.466314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:11.466353 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:59:11.467892 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:11.467902 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:11.468467 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:11.468873 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:11.470499 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:11.471947 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:11.473396 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:11.474173 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:11.474423 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:11.477311 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:11.477320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:11.477339 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:11.477393 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:11.480541 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:11.481420 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:11.481428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:11.481436 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:11.481635 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:11.482071 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:11.482076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:11.482081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:11.482675 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:11.483044 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:11.483518 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 16:59:11.483983 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":212,"threads_started":1}
I20260619 16:59:11.486013 31873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:11.571193 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 31750
I20260619 16:59:11.590545 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 31881
I20260619 16:59:11.610225 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:11.591654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.16.48.3:45753)
0619 16:59:11.592260 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:11.592274 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:11.592316 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:59:11.593948 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:11.593958 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:11.594413 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:11.594807 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:11.596465 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:11.597750 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:11.599203 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:11.599948 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:11.600203 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:11.603057 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:11.603066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:11.603085 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:11.603132 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:11.606435 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:11.607279 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:11.607287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:11.607295 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:11.607499 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:11.607918 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:11.607923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:11.607926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:11.608611 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:11.608942 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:11.609432 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 16:59:11.609909 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":236,"threads_started":1}
I20260619 16:59:11.611816 32004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:11.697609 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 31881
2026-06-19T16:59:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24897 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:59:11 dist-test-slave-3ffd krb5kdc[32067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:59:11 dist-test-slave-3ffd krb5kdc[32067](info): set up 2 sockets
Jun 19 16:59:11 dist-test-slave-3ffd krb5kdc[32067](info): commencing operation
krb5kdc: starting...
W20260619 16:59:12.822161 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 16:59:12 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 16:59:13.084939 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42589
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:42589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:59:13.440018 32080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:59:13.440492 32080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:59:13.440696 32080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:59:13.448621 32080 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:59:13.448784 32080 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:59:13.448876 32080 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:59:13.448971 32080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:59:13.449076 32080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:59:13.449203 32080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:59:13.449294 32080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:59:13.449374 32080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:59:13.459968 32080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:42589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:59:13.461439 32080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:59:13.463086 32080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:59:13.476485 32086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:13.476475 32085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:13.476485 32088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:59:13.478240 32080 server_base.cc:1034] running on GCE node
I20260619 16:59:13.479094 32080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:59:13.480788 32080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:59:13.482035 32080 hybrid_clock.cc:648] HybridClock initialized: now 1781888353481973 us; error 35 us; skew 500 ppm
I20260619 16:59:13.482378 32080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:59:13 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888353, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:59:13.486027 32080 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:59:13.487838 32080 webserver.cc:460] Webserver started at http://127.16.48.62:32923/ using document root <none> and password file <none>
I20260619 16:59:13.488698 32080 fs_manager.cc:359] Metadata directory not provided
I20260619 16:59:13.488849 32080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:59:13.489208 32080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:59:13.492542 32080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c"
format_stamp: "Formatted at 2026-06-19 16:59:13 on dist-test-slave-3ffd"
server_key: "835bdcfa1aaf68fd66ea0b3641dca61f"
server_key_iv: "026370113ea06e7a0e25e60ffb30f30a"
server_key_version: "encryptionkey@0"
I20260619 16:59:13.493443 32080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c"
format_stamp: "Formatted at 2026-06-19 16:59:13 on dist-test-slave-3ffd"
server_key: "835bdcfa1aaf68fd66ea0b3641dca61f"
server_key_iv: "026370113ea06e7a0e25e60ffb30f30a"
server_key_version: "encryptionkey@0"
I20260619 16:59:13.499270 32080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:59:13.503170 32095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:13.504792 32080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:59:13.505018 32080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c"
format_stamp: "Formatted at 2026-06-19 16:59:13 on dist-test-slave-3ffd"
server_key: "835bdcfa1aaf68fd66ea0b3641dca61f"
server_key_iv: "026370113ea06e7a0e25e60ffb30f30a"
server_key_version: "encryptionkey@0"
I20260619 16:59:13.505295 32080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:59:13.539598 32080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:59:13.544106 32080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:59:13.544497 32080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:59:13.565287 32080 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:42589
I20260619 16:59:13.565313 32146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:42589 every 8 connection(s)
I20260619 16:59:13.566890 32080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:59:13.571326 32147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:59:13.571655 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 32080
I20260619 16:59:13.571887 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 16:59:13.572813 16576 external_mini_cluster.cc:1428] Setting key a971f6d0308542d74cc0211c6bf68c35
I20260619 16:59:13.587919 32147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Bootstrap starting.
Jun 19 16:59:13 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:59:13.592170 32147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Neither blocks nor log segments found. Creating new log.
I20260619 16:59:13.593345 32147 log.cc:824] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:59:13.596904 32147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: No bootstrap required, opened a new log
I20260619 16:59:13.601409 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:13.575201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.16.48.62:42589)
0619 16:59:13.576104 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:13.576174 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:13.576236 (+ 62us) server_negotiation.cc:407] Connection header received
0619 16:59:13.578900 (+ 2664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:13.578934 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:13.579465 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:13.579865 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:13.581970 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:13.583369 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:13.585590 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:13.586434 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:13.586597 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:13.591937 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:13.591948 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:13.591965 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:13.592014 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:13.595956 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:13.596941 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:13.596953 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:13.596963 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:13.597208 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:13.597679 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:13.597684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:13.597687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:13.598477 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:13.598787 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:13.599268 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 16:59:13.599795 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":438,"threads_started":1}
I20260619 16:59:13.607275 32147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 16:59:13.607774 32147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:59:13.607894 32147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6119b6ac2a4a6dbad4fd57fc35ba5c, State: Initialized, Role: FOLLOWER
I20260619 16:59:13.608701 32147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 16:59:13.609019 32147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:59:13.609179 32147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:59:13.609391 32147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:59:13.611756 32147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 16:59:13.612437 32147 leader_election.cc:302] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c6119b6ac2a4a6dbad4fd57fc35ba5c; no voters:
I20260619 16:59:13.613273 32147 leader_election.cc:288] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:59:13.613395 32152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:59:13.614305 32152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 1 LEADER]: Becoming Leader. State: Replica: 6c6119b6ac2a4a6dbad4fd57fc35ba5c, State: Running, Role: LEADER
I20260619 16:59:13.614970 32152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 16:59:13.615825 32147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:59:13.618314 32153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } } }
I20260619 16:59:13.618716 32153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: This master's current role is: LEADER
I20260619 16:59:13.618966 32154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c6119b6ac2a4a6dbad4fd57fc35ba5c. Latest consensus state: current_term: 1 leader_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } } }
I20260619 16:59:13.619323 32154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: This master's current role is: LEADER
I20260619 16:59:13.620805 32158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:59:13.627391 32158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:59:13.638068 32158 catalog_manager.cc:1347] Generated new cluster ID: edc6c2f572324e93828489e0bdc92058
I20260619 16:59:13.638208 32158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:59:13.659363 32158 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:59:13.661298 32158 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:59:13.668434 32158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Generated new TSK 0
I20260619 16:59:13.669395 32158 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 16:59:13.736152 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42589
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:59:14.099228 32173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:59:14.099654 32173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:59:14.099823 32173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:59:14.107697 32173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:59:14.107899 32173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:59:14.108080 32173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:59:14.119079 32173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:59:14.120713 32173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:59:14.122192 32173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:59:14.136715 32179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:14.136741 32178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:14.136772 32181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:59:14.137604 32173 server_base.cc:1034] running on GCE node
I20260619 16:59:14.138324 32173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:59:14.139357 32173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:59:14.140609 32173 hybrid_clock.cc:648] HybridClock initialized: now 1781888354140524 us; error 68 us; skew 500 ppm
I20260619 16:59:14.141027 32173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:59:14 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:59:14.144829 32173 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:59:14.146515 32173 webserver.cc:460] Webserver started at http://127.16.48.1:38169/ using document root <none> and password file <none>
I20260619 16:59:14.147262 32173 fs_manager.cc:359] Metadata directory not provided
I20260619 16:59:14.147372 32173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:59:14.147646 32173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:59:14.150884 32173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "c1bc6ec497674e8cb3558d50ebf7b74f"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "0881694c21e7145406f9f6a1ef61b009"
server_key_iv: "e6a4731943fb971372ea1cd1f22fba9f"
server_key_version: "encryptionkey@0"
I20260619 16:59:14.151719 32173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "c1bc6ec497674e8cb3558d50ebf7b74f"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "0881694c21e7145406f9f6a1ef61b009"
server_key_iv: "e6a4731943fb971372ea1cd1f22fba9f"
server_key_version: "encryptionkey@0"
I20260619 16:59:14.157594 32173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:59:14.161783 32188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:14.163595 32173 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:59:14.163822 32173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "c1bc6ec497674e8cb3558d50ebf7b74f"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "0881694c21e7145406f9f6a1ef61b009"
server_key_iv: "e6a4731943fb971372ea1cd1f22fba9f"
server_key_version: "encryptionkey@0"
I20260619 16:59:14.164055 32173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:59:14.198581 32173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:59:14.203553 32173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:59:14.203891 32173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:59:14.205070 32173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:59:14.206591 32173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:59:14.206683 32173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:14.206828 32173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:59:14.206894 32173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:14.242686 32173 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:36761
I20260619 16:59:14.242725 32300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:36761 every 8 connection(s)
I20260619 16:59:14.244091 32173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:59:14.245555 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 32173
I20260619 16:59:14.245800 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 16:59:14.246373 16576 external_mini_cluster.cc:1428] Setting key 22ab43660bcd3e7e2cd3dc8bc54b9a23
Jun 19 16:59:14 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:59:14.268142 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:14.247074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:46067 (local address 127.16.48.62:42589)
0619 16:59:14.247512 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:14.247517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:14.248843 (+ 1326us) server_negotiation.cc:407] Connection header received
0619 16:59:14.250806 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:14.250814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:14.250926 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:14.251057 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:14.252797 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:14.254004 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:14.255702 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:14.256381 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:14.256491 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:14.261241 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:14.261247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:14.261250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:14.261291 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:14.264326 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:14.265237 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:14.265251 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:14.265253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:14.265389 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:14.265908 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:14.265912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:14.265914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:14.266215 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:14.266392 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:14.267290 (+ 898us) server_negotiation.cc:299] Negotiation successful
0619 16:59:14.267602 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":215,"threads_started":1}
I20260619 16:59:14.268909 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:14.247782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:46067)
0619 16:59:14.248660 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:14.248713 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:14.250480 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:14.251277 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:14.251286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:14.251894 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:59:14.252595 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:14.252615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:14.254193 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:14.254197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:14.255493 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:14.255506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:14.256672 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:14.256682 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:14.257050 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:14.257929 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:59:14.257966 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:59:14.260921 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:59:14.264523 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:14.264534 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:14.264562 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:14.265056 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:14.265521 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:14.265526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:14.265530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:14.265764 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:14.266362 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:59:14.266371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:59:14.266776 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 16:59:14.267083 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 16:59:14.267582 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":392,"threads_started":1}
I20260619 16:59:14.270773 32301 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42589
I20260619 16:59:14.271153 32301 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 16:59:14.272176 32301 heartbeater.cc:511] Master 127.16.48.62:42589 requested a full tablet report, sending...
I20260619 16:59:14.274588 32112 ts_manager.cc:194] Registered new tserver with Master: c1bc6ec497674e8cb3558d50ebf7b74f (127.16.48.1:36761)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 16:59:14.276674 32112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:46067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 16:59:14.304497 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42589
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:59:14.662082 32307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:59:14.662459 32307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:59:14.662643 32307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:59:14.670838 32307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:59:14.671023 32307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:59:14.671221 32307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:59:14.681563 32307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:59:14.683046 32307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:59:14.684514 32307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:59:14.699445 32312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:14.699723 32313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:14.700590 32315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:59:14.701867 32307 server_base.cc:1034] running on GCE node
I20260619 16:59:14.702992 32307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:59:14.704578 32307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:59:14.705855 32307 hybrid_clock.cc:648] HybridClock initialized: now 1781888354705794 us; error 46 us; skew 500 ppm
I20260619 16:59:14.706262 32307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:59:14 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:59:14.710101 32307 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:59:14.711889 32307 webserver.cc:460] Webserver started at http://127.16.48.2:36517/ using document root <none> and password file <none>
I20260619 16:59:14.712739 32307 fs_manager.cc:359] Metadata directory not provided
I20260619 16:59:14.712896 32307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:59:14.713233 32307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:59:14.716497 32307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "831f0f39690d4031ae7dc05ce7938625"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "858ec586717582799a321bd633174c47"
server_key_iv: "f6d78fbdbb407f64e431b86bb007cfd1"
server_key_version: "encryptionkey@0"
I20260619 16:59:14.717393 32307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "831f0f39690d4031ae7dc05ce7938625"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "858ec586717582799a321bd633174c47"
server_key_iv: "f6d78fbdbb407f64e431b86bb007cfd1"
server_key_version: "encryptionkey@0"
I20260619 16:59:14.723295 32307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:59:14.727432 32322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:14.729319 32307 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 16:59:14.729511 32307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "831f0f39690d4031ae7dc05ce7938625"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "858ec586717582799a321bd633174c47"
server_key_iv: "f6d78fbdbb407f64e431b86bb007cfd1"
server_key_version: "encryptionkey@0"
I20260619 16:59:14.729768 32307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:59:14.760507 32307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:59:14.765390 32307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:59:14.765659 32307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:59:14.766624 32307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:59:14.768111 32307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:59:14.768244 32307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 16:59:14.768338 32307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:59:14.768432 32307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:14.803493 32307 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:43961
I20260619 16:59:14.803544 32434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:43961 every 8 connection(s)
I20260619 16:59:14.804877 32307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:59:14.810050 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 32307
I20260619 16:59:14.810313 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 16:59:14.810865 16576 external_mini_cluster.cc:1428] Setting key afa4efac5b5fa853b01831fc193d666d
Jun 19 16:59:14 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:59:14.827229 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:14.807569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:41537 (local address 127.16.48.62:42589)
0619 16:59:14.808104 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:14.808109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:14.809250 (+ 1141us) server_negotiation.cc:407] Connection header received
0619 16:59:14.811467 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:14.811473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:14.811582 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:14.811717 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:14.813393 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:14.814136 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:14.815781 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:14.816348 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:14.816513 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:14.820565 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:14.820575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:14.820579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:14.820615 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:14.823426 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:14.824567 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:14.824572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:14.824575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:14.824722 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:14.825288 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:14.825292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:14.825294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:14.825597 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:14.825722 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:14.826713 (+ 991us) server_negotiation.cc:299] Negotiation successful
0619 16:59:14.826921 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":268,"threads_started":1}
I20260619 16:59:14.828979 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:14.808299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:41537)
0619 16:59:14.809056 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:14.809095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:14.811160 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:14.811981 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:14.811989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:14.812630 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:59:14.813220 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:14.813239 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:14.814310 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:14.814314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:14.815616 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:14.815631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:14.816632 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:14.816640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:14.816889 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:14.817695 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:59:14.817721 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:59:14.820279 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:59:14.823627 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:14.823640 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:14.823664 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:14.824421 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:14.824844 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:14.824852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:14.824858 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:14.825165 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:14.825738 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:59:14.825750 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:59:14.826296 (+ 546us) client_negotiation.cc:769] Sending connection context
0619 16:59:14.826622 (+ 326us) client_negotiation.cc:240] Negotiation successful
0619 16:59:14.827282 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":313,"threads_started":1}
I20260619 16:59:14.830739 32435 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42589
I20260619 16:59:14.831101 32435 heartbeater.cc:464] Registering TS with master...
I20260619 16:59:14.831904 32435 heartbeater.cc:511] Master 127.16.48.62:42589 requested a full tablet report, sending...
I20260619 16:59:14.833729 32112 ts_manager.cc:194] Registered new tserver with Master: 831f0f39690d4031ae7dc05ce7938625 (127.16.48.2:43961)
I20260619 16:59:14.834790 32112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:41537
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 16:59:14.869567 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42589
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:59:15.217053 32441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:59:15.217433 32441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:59:15.217634 32441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:59:15.225244 32441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:59:15.225366 32441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:59:15.225557 32441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:59:15.236346 32441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:59:15.237739 32441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:59:15.239172 32441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:59:15.253684 32446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:15.253647 32447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:15.253667 32449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:59:15.254459 32441 server_base.cc:1034] running on GCE node
I20260619 16:59:15.255216 32441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:59:15.256304 32441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:59:15.257536 32441 hybrid_clock.cc:648] HybridClock initialized: now 1781888355257473 us; error 63 us; skew 500 ppm
I20260619 16:59:15.257870 32441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:59:15 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888355, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:59:15.261272 32441 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:59:15.262773 32441 webserver.cc:460] Webserver started at http://127.16.48.3:40761/ using document root <none> and password file <none>
I20260619 16:59:15.263561 32441 fs_manager.cc:359] Metadata directory not provided
I20260619 16:59:15.263669 32441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:59:15.263926 32441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:59:15.267190 32441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "f29b94a9f30949d88a5a3d88f48a537e"
format_stamp: "Formatted at 2026-06-19 16:59:15 on dist-test-slave-3ffd"
server_key: "b832add1e678ca46fa331f2f608b3ad6"
server_key_iv: "d5e9114f135429ceab7903d53474dd0c"
server_key_version: "encryptionkey@0"
I20260619 16:59:15.267969 32441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "f29b94a9f30949d88a5a3d88f48a537e"
format_stamp: "Formatted at 2026-06-19 16:59:15 on dist-test-slave-3ffd"
server_key: "b832add1e678ca46fa331f2f608b3ad6"
server_key_iv: "d5e9114f135429ceab7903d53474dd0c"
server_key_version: "encryptionkey@0"
I20260619 16:59:15.274053 32441 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:59:15.277961 32456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:15.279497 32441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:59:15.279660 32441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "f29b94a9f30949d88a5a3d88f48a537e"
format_stamp: "Formatted at 2026-06-19 16:59:15 on dist-test-slave-3ffd"
server_key: "b832add1e678ca46fa331f2f608b3ad6"
server_key_iv: "d5e9114f135429ceab7903d53474dd0c"
server_key_version: "encryptionkey@0"
I20260619 16:59:15.279844 32441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:59:15.280273 32301 heartbeater.cc:503] Master 127.16.48.62:42589 was elected leader, sending a full tablet report...
I20260619 16:59:15.302577 32441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:59:15.307252 32441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:59:15.307579 32441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:59:15.308719 32441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:59:15.310300 32441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:59:15.310410 32441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:15.310526 32441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:59:15.310576 32441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:15.345832 32441 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:32841
I20260619 16:59:15.345880 32568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:32841 every 8 connection(s)
I20260619 16:59:15.347237 32441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:59:15.355252 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 32441
I20260619 16:59:15.355473 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 16:59:15.356022 16576 external_mini_cluster.cc:1428] Setting key 921887fbcc52e06cd01935054aa110fc
Jun 19 16:59:15 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888355, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:59:15.369891 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.350205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:60981 (local address 127.16.48.62:42589)
0619 16:59:15.350708 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.350713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.351622 (+ 909us) server_negotiation.cc:407] Connection header received
0619 16:59:15.353679 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.353685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.353808 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.353938 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:15.355786 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.356761 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.359115 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.359825 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.359947 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.363672 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:15.363678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:15.363681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:15.363718 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:15.366551 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:15.367457 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:15.367462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:15.367464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:15.367596 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:15.368022 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:15.368026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:15.368028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:15.368464 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:15.368594 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.369344 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.369576 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":261,"threads_started":1}
I20260619 16:59:15.371176 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.350659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:60981)
0619 16:59:15.351419 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:15.351466 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:15.353345 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:15.354216 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:15.354225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:15.354820 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:59:15.355566 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.355602 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.356962 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.356967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.358911 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.358928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.360105 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.360111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.360425 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.361153 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:59:15.361184 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:59:15.363416 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:59:15.366732 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:15.366740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:15.366755 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:15.367290 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:15.367713 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:15.367717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:15.367719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:15.367911 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:15.368599 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:59:15.368608 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:59:15.369004 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 16:59:15.369254 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 16:59:15.369739 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":309,"threads_started":1}
I20260619 16:59:15.372721 32569 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42589
I20260619 16:59:15.373042 32569 heartbeater.cc:464] Registering TS with master...
I20260619 16:59:15.373782 32569 heartbeater.cc:511] Master 127.16.48.62:42589 requested a full tablet report, sending...
I20260619 16:59:15.375402 32111 ts_manager.cc:194] Registered new tserver with Master: f29b94a9f30949d88a5a3d88f48a537e (127.16.48.3:32841)
I20260619 16:59:15.376405 32111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:60981
I20260619 16:59:15.385260 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:59:15.404237 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.389976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.16.48.62:42589)
0619 16:59:15.390347 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.390354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.390634 (+ 280us) server_negotiation.cc:407] Connection header received
0619 16:59:15.390885 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.390889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.390957 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.391066 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:15.392583 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.393347 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.394741 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.395481 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.395620 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.398306 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:15.398312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:15.398315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:15.398353 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:15.401158 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:15.402020 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:15.402026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:15.402030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:15.402189 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:15.402628 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:15.402633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:15.402636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:15.402988 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:15.403271 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.403598 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.403852 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260619 16:59:15.408685 32111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:59:15.416260 32111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:59:15.442721 32583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.430161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:32841 (local address 127.0.0.1:55320)
0619 16:59:15.431574 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:15.431626 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:15.431822 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:15.433986 (+ 2164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:15.433993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:15.434022 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:15.434467 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.434484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.436827 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.436832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.439226 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.439240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.441331 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.441339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.441649 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.441723 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 16:59:15.441844 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:59:15.442067 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":1,"thread_start_us":2873,"threads_started":1}
I20260619 16:59:15.443672 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.430222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.16.48.3:32841)
0619 16:59:15.433255 (+ 3033us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.433263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.433301 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:59:15.433414 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.433425 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.433743 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.434685 (+ 942us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:15.434996 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.436664 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.439459 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.441082 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.442049 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.442324 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.442403 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.442595 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2857,"thread_start_us":2692,"threads_started":1}
I20260619 16:59:15.445873 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.430887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.16.48.1:36761)
0619 16:59:15.432878 (+ 1991us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.432889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.433491 (+ 602us) server_negotiation.cc:407] Connection header received
0619 16:59:15.433792 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.433800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.434043 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.434233 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:15.436692 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.438346 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.443200 (+ 4854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.444832 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.445002 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.445275 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.445480 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.445625 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1800,"thread_start_us":1633,"threads_started":1}
I20260619 16:59:15.446628 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.430246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:36761 (local address 127.0.0.1:57188)
0619 16:59:15.433372 (+ 3126us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:15.433392 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:15.433598 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:15.435966 (+ 2368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:15.435970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:15.435997 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:15.436465 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.436476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.439893 (+ 3417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.439898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.442984 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.442998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.445050 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.445056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.445259 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.445296 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:59:15.446168 (+ 872us) client_negotiation.cc:240] Negotiation successful
0619 16:59:15.446308 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3003,"mutex_wait_us":881,"thread_start_us":932,"threads_started":1}
I20260619 16:59:15.448531 32504 tablet_service.cc:1467] Processing CreateTablet for tablet 2720eb20f223468a813b1e9247842cf1 (DEFAULT_TABLE table=test-table [id=175402514e174d1d972e0bb048e2ca57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:59:15.450173 32504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2720eb20f223468a813b1e9247842cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:59:15.451058 32236 tablet_service.cc:1467] Processing CreateTablet for tablet 2720eb20f223468a813b1e9247842cf1 (DEFAULT_TABLE table=test-table [id=175402514e174d1d972e0bb048e2ca57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:59:15.452355 32236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2720eb20f223468a813b1e9247842cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:59:15.455022 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.433578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:43961 (local address 127.0.0.1:34772)
0619 16:59:15.442467 (+ 8889us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:15.442489 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:15.442642 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:15.443796 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:15.443802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:15.443830 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:15.444213 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.444227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.446126 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.446134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.448269 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.448288 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.454090 (+ 5802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.454098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.454307 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.454348 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:59:15.454459 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:59:15.454630 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8718,"thread_start_us":8557,"threads_started":1}
I20260619 16:59:15.455930 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.433843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.16.48.2:43961)
0619 16:59:15.441922 (+ 8079us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.441932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.442780 (+ 848us) server_negotiation.cc:407] Connection header received
0619 16:59:15.442902 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.442910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.443302 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.443507 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:15.444397 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.445935 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.452267 (+ 6332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.453817 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.453982 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.454186 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.454561 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.455566 (+ 1005us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7906,"thread_start_us":7784,"threads_started":1}
I20260619 16:59:15.460330 32370 tablet_service.cc:1467] Processing CreateTablet for tablet 2720eb20f223468a813b1e9247842cf1 (DEFAULT_TABLE table=test-table [id=175402514e174d1d972e0bb048e2ca57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:59:15.461926 32370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2720eb20f223468a813b1e9247842cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:59:15.464565 32588 tablet_bootstrap.cc:492] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: Bootstrap starting.
I20260619 16:59:15.466845 32589 tablet_bootstrap.cc:492] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: Bootstrap starting.
I20260619 16:59:15.468879 32588 tablet_bootstrap.cc:654] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: Neither blocks nor log segments found. Creating new log.
I20260619 16:59:15.470320 32588 log.cc:824] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:59:15.470894 32589 tablet_bootstrap.cc:654] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: Neither blocks nor log segments found. Creating new log.
I20260619 16:59:15.472409 32589 log.cc:824] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:59:15.474177 32588 tablet_bootstrap.cc:492] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: No bootstrap required, opened a new log
I20260619 16:59:15.474489 32588 ts_tablet_manager.cc:1397] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 16:59:15.476118 32589 tablet_bootstrap.cc:492] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: No bootstrap required, opened a new log
I20260619 16:59:15.476491 32589 ts_tablet_manager.cc:1397] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 16:59:15.478396 32592 tablet_bootstrap.cc:492] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: Bootstrap starting.
I20260619 16:59:15.482707 32592 tablet_bootstrap.cc:654] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: Neither blocks nor log segments found. Creating new log.
I20260619 16:59:15.485344 32588 raft_consensus.cc:348] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } }
I20260619 16:59:15.485885 32588 raft_consensus.cc:374] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:59:15.486038 32588 raft_consensus.cc:729] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29b94a9f30949d88a5a3d88f48a537e, State: Initialized, Role: FOLLOWER
I20260619 16:59:15.486347 32592 log.cc:824] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: Log is configured to *not* fsync() on all Append() calls
I20260619 16:59:15.486045 32589 raft_consensus.cc:348] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } }
I20260619 16:59:15.486562 32589 raft_consensus.cc:374] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:59:15.486691 32589 raft_consensus.cc:729] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1bc6ec497674e8cb3558d50ebf7b74f, State: Initialized, Role: FOLLOWER
I20260619 16:59:15.486763 32588 consensus_queue.cc:260] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } }
I20260619 16:59:15.487367 32589 consensus_queue.cc:260] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } }
I20260619 16:59:15.488381 32569 heartbeater.cc:503] Master 127.16.48.62:42589 was elected leader, sending a full tablet report...
I20260619 16:59:15.491231 32588 ts_tablet_manager.cc:1428] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260619 16:59:15.493571 32589 ts_tablet_manager.cc:1428] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260619 16:59:15.495923 32592 tablet_bootstrap.cc:492] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: No bootstrap required, opened a new log
I20260619 16:59:15.496341 32592 ts_tablet_manager.cc:1397] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.011s
W20260619 16:59:15.497165 32302 tablet.cc:2369] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:59:15.503856 32592 raft_consensus.cc:348] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } }
I20260619 16:59:15.504379 32592 raft_consensus.cc:374] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:59:15.504482 32592 raft_consensus.cc:729] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831f0f39690d4031ae7dc05ce7938625, State: Initialized, Role: FOLLOWER
I20260619 16:59:15.505177 32592 consensus_queue.cc:260] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } }
I20260619 16:59:15.506297 32435 heartbeater.cc:503] Master 127.16.48.62:42589 was elected leader, sending a full tablet report...
I20260619 16:59:15.506973 32592 ts_tablet_manager.cc:1428] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20260619 16:59:15.557112 32436 tablet.cc:2369] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:59:15.599189 32570 tablet.cc:2369] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:59:15.613430 32594 raft_consensus.cc:482] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:59:15.613833 32594 raft_consensus.cc:504] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } }
I20260619 16:59:15.615569 32594 leader_election.cc:288] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1bc6ec497674e8cb3558d50ebf7b74f (127.16.48.1:36761), 831f0f39690d4031ae7dc05ce7938625 (127.16.48.2:43961)
I20260619 16:59:15.621626 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.615887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:36761 (local address 127.16.48.3:33979)
0619 16:59:15.616285 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:15.616308 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:15.616469 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:15.616832 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:15.616836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:15.616871 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:15.617086 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.617094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.618248 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.618255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.619573 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.619586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.620817 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.620823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.621008 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.621034 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:59:15.621147 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:59:15.621288 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260619 16:59:15.621628 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.616233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:33979 (local address 127.16.48.1:36761)
0619 16:59:15.616470 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.616476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.616498 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:59:15.616611 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.616618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.616689 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.616810 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:15.617239 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.618092 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.619765 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.620641 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.620796 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.620902 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.621181 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.621379 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 16:59:15.622424 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.616054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:39027 (local address 127.16.48.2:43961)
0619 16:59:15.616287 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.616293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.617073 (+ 780us) server_negotiation.cc:407] Connection header received
0619 16:59:15.617307 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.617312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.617382 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.617481 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:15.618002 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.618717 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.620521 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.621351 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.621524 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.621627 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.622015 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.622155 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:59:15.622632 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.615887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:43961 (local address 127.16.48.3:39027)
0619 16:59:15.616933 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:15.616948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:15.617131 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:15.617494 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:15.617499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:15.617544 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:15.617841 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.617850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.618882 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.618886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.620357 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:15.620369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.621588 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.621594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:15.621833 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.621880 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 16:59:15.621985 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:59:15.622151 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":345,"thread_start_us":366,"threads_started":1}
I20260619 16:59:15.623317 32256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2720eb20f223468a813b1e9247842cf1" candidate_uuid: "f29b94a9f30949d88a5a3d88f48a537e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" is_pre_election: true
I20260619 16:59:15.623839 32390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2720eb20f223468a813b1e9247842cf1" candidate_uuid: "f29b94a9f30949d88a5a3d88f48a537e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "831f0f39690d4031ae7dc05ce7938625" is_pre_election: true
I20260619 16:59:15.624022 32256 raft_consensus.cc:2454] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b94a9f30949d88a5a3d88f48a537e in term 0.
I20260619 16:59:15.624426 32390 raft_consensus.cc:2454] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b94a9f30949d88a5a3d88f48a537e in term 0.
I20260619 16:59:15.624832 32457 leader_election.cc:302] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1bc6ec497674e8cb3558d50ebf7b74f, f29b94a9f30949d88a5a3d88f48a537e; no voters:
I20260619 16:59:15.625387 32594 raft_consensus.cc:2781] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:59:15.625593 32594 raft_consensus.cc:482] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:59:15.625720 32594 raft_consensus.cc:3037] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:59:15.627952 32594 raft_consensus.cc:504] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } }
I20260619 16:59:15.628679 32594 leader_election.cc:288] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [CANDIDATE]: Term 1 election: Requested vote from peers c1bc6ec497674e8cb3558d50ebf7b74f (127.16.48.1:36761), 831f0f39690d4031ae7dc05ce7938625 (127.16.48.2:43961)
I20260619 16:59:15.629308 32390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2720eb20f223468a813b1e9247842cf1" candidate_uuid: "f29b94a9f30949d88a5a3d88f48a537e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "831f0f39690d4031ae7dc05ce7938625"
I20260619 16:59:15.629309 32256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2720eb20f223468a813b1e9247842cf1" candidate_uuid: "f29b94a9f30949d88a5a3d88f48a537e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f"
I20260619 16:59:15.629549 32256 raft_consensus.cc:3037] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:59:15.629590 32390 raft_consensus.cc:3037] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:59:15.632951 32390 raft_consensus.cc:2454] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b94a9f30949d88a5a3d88f48a537e in term 1.
I20260619 16:59:15.632961 32256 raft_consensus.cc:2454] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b94a9f30949d88a5a3d88f48a537e in term 1.
I20260619 16:59:15.633570 32459 leader_election.cc:302] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 831f0f39690d4031ae7dc05ce7938625, c1bc6ec497674e8cb3558d50ebf7b74f, f29b94a9f30949d88a5a3d88f48a537e; no voters:
I20260619 16:59:15.634021 32594 raft_consensus.cc:2781] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:59:15.634538 32594 raft_consensus.cc:686] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 1 LEADER]: Becoming Leader. State: Replica: f29b94a9f30949d88a5a3d88f48a537e, State: Running, Role: LEADER
I20260619 16:59:15.635130 32594 consensus_queue.cc:237] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } }
I20260619 16:59:15.639879 32111 catalog_manager.cc:5495] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e reported cstate change: term changed from 0 to 1, leader changed from <none> to f29b94a9f30949d88a5a3d88f48a537e (127.16.48.3). New cstate: current_term: 1 leader_uuid: "f29b94a9f30949d88a5a3d88f48a537e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } health_report { overall_health: UNKNOWN } } }
I20260619 16:59:15.685307 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.678825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.16.48.3:32841)
0619 16:59:15.679063 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.679069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.679094 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:59:15.679254 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.679259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.679353 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.679533 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:59:15.679992 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.680863 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.682592 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.683067 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.683237 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.683838 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:59:15.684544 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:59:15.684722 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.684835 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.684943 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:59:15.694774 32390 raft_consensus.cc:1261] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [term 1 FOLLOWER]: Refusing update from remote peer f29b94a9f30949d88a5a3d88f48a537e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:59:15.694774 32256 raft_consensus.cc:1261] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [term 1 FOLLOWER]: Refusing update from remote peer f29b94a9f30949d88a5a3d88f48a537e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:59:15.696018 32598 consensus_queue.cc:1035] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:59:15.696478 32594 consensus_queue.cc:1035] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [LEADER]: Connected to new peer: Peer: permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:59:15.708252 32601 mvcc.cc:205] Tried to move back new op lower bound from 7298614704912875520 to 7298614704685826048. Current Snapshot: MvccSnapshot[applied={T|T < 7298614704912875520}]
I20260619 16:59:15.712744 32602 mvcc.cc:205] Tried to move back new op lower bound from 7298614704912875520 to 7298614704685826048. Current Snapshot: MvccSnapshot[applied={T|T < 7298614704912875520}]
I20260619 16:59:15.724859 32112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:59:15.725204 32112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:59:15.729355 32112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Sending DeleteTablet for 3 replicas of tablet 2720eb20f223468a813b1e9247842cf1
I20260619 16:59:15.730809 32236 tablet_service.cc:1514] Processing DeleteTablet for tablet 2720eb20f223468a813b1e9247842cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:59:15 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:57188
I20260619 16:59:15.731040 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 32173
I20260619 16:59:15.731128 32504 tablet_service.cc:1514] Processing DeleteTablet for tablet 2720eb20f223468a813b1e9247842cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:59:15 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:55320
I20260619 16:59:15.731938 32370 tablet_service.cc:1514] Processing DeleteTablet for tablet 2720eb20f223468a813b1e9247842cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:59:15 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:34772
I20260619 16:59:15.746061 32613 tablet_replica.cc:330] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: stopping tablet replica
I20260619 16:59:15.746876 32613 raft_consensus.cc:2229] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:59:15.747618 32613 raft_consensus.cc:2258] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:59:15.749333 32611 tablet_replica.cc:330] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: stopping tablet replica
I20260619 16:59:15.749876 32611 raft_consensus.cc:2229] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:59:15.750314 32610 tablet_replica.cc:330] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: stopping tablet replica
I20260619 16:59:15.750725 32611 raft_consensus.cc:2258] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:59:15.751531 32613 ts_tablet_manager.cc:1905] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:59:15.752974 32610 raft_consensus.cc:2229] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:59:15.753123 32611 ts_tablet_manager.cc:1905] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:59:15.755618 32610 raft_consensus.cc:2258] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:59:15.758044 32610 ts_tablet_manager.cc:1905] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:59:15.759786 32613 ts_tablet_manager.cc:1918] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:59:15.760066 32613 log.cc:1196] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/2720eb20f223468a813b1e9247842cf1
I20260619 16:59:15.760838 32613 ts_tablet_manager.cc:1939] T 2720eb20f223468a813b1e9247842cf1 P 831f0f39690d4031ae7dc05ce7938625: Deleting consensus metadata
I20260619 16:59:15.762305 32611 ts_tablet_manager.cc:1918] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:59:15.762627 32611 log.cc:1196] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/2720eb20f223468a813b1e9247842cf1
I20260619 16:59:15.763231 32098 catalog_manager.cc:4841] TS 831f0f39690d4031ae7dc05ce7938625 (127.16.48.2:43961): tablet 2720eb20f223468a813b1e9247842cf1 (table test-table [id=175402514e174d1d972e0bb048e2ca57]) successfully deleted
I20260619 16:59:15.763372 32611 ts_tablet_manager.cc:1939] T 2720eb20f223468a813b1e9247842cf1 P f29b94a9f30949d88a5a3d88f48a537e: Deleting consensus metadata
I20260619 16:59:15.765494 32098 catalog_manager.cc:4841] TS f29b94a9f30949d88a5a3d88f48a537e (127.16.48.3:32841): tablet 2720eb20f223468a813b1e9247842cf1 (table test-table [id=175402514e174d1d972e0bb048e2ca57]) successfully deleted
Jun 19 16:59:15 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 16:59:15.769516 32610 ts_tablet_manager.cc:1918] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:59:15.769822 32610 log.cc:1196] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/2720eb20f223468a813b1e9247842cf1
I20260619 16:59:15.770427 32610 ts_tablet_manager.cc:1939] T 2720eb20f223468a813b1e9247842cf1 P c1bc6ec497674e8cb3558d50ebf7b74f: Deleting consensus metadata
I20260619 16:59:15.772667 32096 catalog_manager.cc:4841] TS c1bc6ec497674e8cb3558d50ebf7b74f (127.16.48.1:36761): tablet 2720eb20f223468a813b1e9247842cf1 (table test-table [id=175402514e174d1d972e0bb048e2ca57]) successfully deleted
I20260619 16:59:15.776654 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.736213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.16.48.1:36761)
0619 16:59:15.736448 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.736454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.744815 (+ 8361us) server_negotiation.cc:407] Connection header received
0619 16:59:15.745083 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.745089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.745256 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.745403 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:15.751450 (+ 6047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.752617 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.758323 (+ 5706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.759020 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.759146 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.768930 (+ 9784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:15.768945 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:15.768962 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:15.769006 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:15.773113 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:15.774047 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:15.774056 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:15.774059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:15.774224 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:15.774704 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:15.774709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:15.774711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:15.775123 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:15.775457 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.775876 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.776213 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 16:59:15.777396 32296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:15.905138 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 32173
I20260619 16:59:15.923744 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 32307
Jun 19 16:59:15 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 16:59:15.945072 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:15.924627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.16.48.2:43961)
0619 16:59:15.924814 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:15.924818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:15.924836 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:59:15.925071 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:15.925075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:15.925144 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:15.925252 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:15.927289 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.928044 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.930517 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:15.930950 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:15.931063 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:15.937396 (+ 6333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:15.937422 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:15.937445 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:15.937508 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:15.941707 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:15.942629 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:15.942640 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:15.942644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:15.942817 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:15.943192 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:15.943197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:15.943201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:15.943732 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:15.944063 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:15.944360 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 16:59:15.944655 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:59:15.945909 32430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:16.039561 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 32307
I20260619 16:59:16.057806 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 32441
Jun 19 16:59:16 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 16:59:16.077628 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:16.058775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.16.48.3:32841)
0619 16:59:16.058998 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:16.059004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:16.059028 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:59:16.059183 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:16.059189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:16.059305 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:16.059437 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:16.060883 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:16.062090 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:16.063494 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:16.064315 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:16.064460 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:16.069452 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:16.069471 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:16.069494 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:16.069551 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:16.073878 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:16.074796 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:16.074806 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:16.074810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:16.075014 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:16.075557 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:16.075563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:16.075566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:16.076074 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:16.076638 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:16.076897 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 16:59:16.077183 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:59:16.078544 32564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:16.204280 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 32441
I20260619 16:59:16.224592 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 32080
I20260619 16:59:16.225777 32142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:16.340027 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 32080
I20260619 16:59:36.362783 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42589
--webserver_interface=127.16.48.62
--webserver_port=32923
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:42589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:59:36.719825 32618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:59:36.720260 32618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:59:36.720424 32618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:59:36.728281 32618 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:59:36.728394 32618 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:59:36.728480 32618 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:59:36.728579 32618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:59:36.728669 32618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:59:36.728793 32618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:59:36.728861 32618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:59:36.728940 32618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:59:36.739673 32618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:42589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=32923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:59:36.741191 32618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:59:36.742677 32618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:59:36.767545 32624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:36.767421 32625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:36.767417 32627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:59:36.768420 32618 server_base.cc:1034] running on GCE node
I20260619 16:59:36.769421 32618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:59:36.771196 32618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:59:36.772472 32618 hybrid_clock.cc:648] HybridClock initialized: now 1781888376772379 us; error 69 us; skew 500 ppm
I20260619 16:59:36.772902 32618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:59:36 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888376, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:59:36.776894 32618 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 16:59:36.778741 32618 webserver.cc:460] Webserver started at http://127.16.48.62:32923/ using document root <none> and password file <none>
I20260619 16:59:36.779613 32618 fs_manager.cc:359] Metadata directory not provided
I20260619 16:59:36.779733 32618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:59:36.786965 32618 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20260619 16:59:36.790416 32634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:36.792037 32618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:59:36.792347 32618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c"
format_stamp: "Formatted at 2026-06-19 16:59:13 on dist-test-slave-3ffd"
server_key: "835bdcfa1aaf68fd66ea0b3641dca61f"
server_key_iv: "026370113ea06e7a0e25e60ffb30f30a"
server_key_version: "encryptionkey@0"
I20260619 16:59:36.793529 32618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:59:36.829689 32618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:59:36.834708 32618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:59:36.835063 32618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:59:36.855829 32618 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:42589
I20260619 16:59:36.855859 32685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:42589 every 8 connection(s)
I20260619 16:59:36.857616 32618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 16:59:36.860081 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 32618
I20260619 16:59:36.861173 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:36761
--local_ip_for_outbound_sockets=127.16.48.1
--tserver_master_addrs=127.16.48.62:42589
--webserver_port=38169
--webserver_interface=127.16.48.1
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:59:36.868896 32686 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:59:36.873378 32686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Bootstrap starting.
I20260619 16:59:36.894721 32686 log.cc:824] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:59:36.907796 32686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:59:36.908542 32686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Bootstrap complete.
I20260619 16:59:36.919181 32686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 16:59:36.920171 32686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6119b6ac2a4a6dbad4fd57fc35ba5c, State: Initialized, Role: FOLLOWER
I20260619 16:59:36.921000 32686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 16:59:36.921324 32686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:59:36.921487 32686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:59:36.921691 32686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:59:36.926180 32686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 16:59:36.926806 32686 leader_election.cc:302] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c6119b6ac2a4a6dbad4fd57fc35ba5c; no voters:
I20260619 16:59:36.927541 32686 leader_election.cc:288] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:59:36.927701 32690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:59:36.929003 32690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 2 LEADER]: Becoming Leader. State: Replica: 6c6119b6ac2a4a6dbad4fd57fc35ba5c, State: Running, Role: LEADER
I20260619 16:59:36.929692 32690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 16:59:36.930109 32686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:59:36.932300 32692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c6119b6ac2a4a6dbad4fd57fc35ba5c. Latest consensus state: current_term: 2 leader_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } } }
I20260619 16:59:36.932729 32692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: This master's current role is: LEADER
I20260619 16:59:36.933334 32691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } } }
I20260619 16:59:36.933645 32691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: This master's current role is: LEADER
I20260619 16:59:36.945853 32695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:59:36.957701 32695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:59:36.959928 32695 catalog_manager.cc:1259] Loaded cluster ID: edc6c2f572324e93828489e0bdc92058
I20260619 16:59:36.960150 32695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:59:36.966148 32695 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:59:36.968644 32695 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Loaded TSK: 0
I20260619 16:59:37.000424 32695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Generated new TSK 1
I20260619 16:59:37.000869 32695 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:59:37.261695 32688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:59:37.262013 32688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:59:37.262171 32688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:59:37.270102 32688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:59:37.270231 32688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:59:37.270416 32688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 16:59:37.281098 32688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:36761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=38169
--tserver_master_addrs=127.16.48.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:59:37.282557 32688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:59:37.284032 32688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:59:37.300354 32714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:37.300371 32713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:37.300668 32716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:59:37.306303 32688 server_base.cc:1034] running on GCE node
I20260619 16:59:37.307126 32688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:59:37.308353 32688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:59:37.309648 32688 hybrid_clock.cc:648] HybridClock initialized: now 1781888377309577 us; error 63 us; skew 500 ppm
I20260619 16:59:37.310061 32688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:59:37 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888377, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:59:37.313858 32688 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 16:59:37.315865 32688 webserver.cc:460] Webserver started at http://127.16.48.1:38169/ using document root <none> and password file <none>
I20260619 16:59:37.316776 32688 fs_manager.cc:359] Metadata directory not provided
I20260619 16:59:37.316970 32688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:59:37.323190 32688 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:59:37.326668 32723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:37.328467 32688 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:59:37.328668 32688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "c1bc6ec497674e8cb3558d50ebf7b74f"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "0881694c21e7145406f9f6a1ef61b009"
server_key_iv: "e6a4731943fb971372ea1cd1f22fba9f"
server_key_version: "encryptionkey@0"
I20260619 16:59:37.329726 32688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:59:37.374794 32688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:59:37.379285 32688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:59:37.379644 32688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:59:37.380826 32688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:59:37.382273 32688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:59:37.382393 32688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:37.382531 32688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:59:37.382629 32688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:37.418473 32688 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:36761
I20260619 16:59:37.418525 367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:36761 every 8 connection(s)
I20260619 16:59:37.420192 32688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 16:59:37.422220 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 32688
I20260619 16:59:37.422984 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:43961
--local_ip_for_outbound_sockets=127.16.48.2
--tserver_master_addrs=127.16.48.62:42589
--webserver_port=36517
--webserver_interface=127.16.48.2
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:59:37 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888377, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:59:37.453610 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:37.423901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:49897)
0619 16:59:37.425026 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:37.425085 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:37.428040 (+ 2955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:37.430092 (+ 2052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:37.430107 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:37.431093 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:59:37.432014 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:37.432041 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:37.435031 (+ 2990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:37.435037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:37.436649 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:37.436671 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:37.438522 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:37.438529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:37.438778 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:37.439645 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:59:37.439673 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:59:37.442381 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:59:37.448505 (+ 6124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:37.448514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:37.448529 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:37.449114 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:37.449855 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:37.449859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:37.449861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:37.450098 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:37.451302 (+ 1204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:59:37.451309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:59:37.451652 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 16:59:37.451908 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 16:59:37.452498 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":444,"threads_started":1}
I20260619 16:59:37.454069 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:37.423652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:49897 (local address 127.16.48.62:42589)
0619 16:59:37.424463 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:37.424488 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:37.425289 (+ 801us) server_negotiation.cc:407] Connection header received
0619 16:59:37.428897 (+ 3608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:37.428914 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:37.429406 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:37.429804 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:37.432420 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:37.434790 (+ 2370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:37.436936 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:37.438186 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:37.438441 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:37.442855 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:37.442871 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:37.442899 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:37.442963 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:37.448254 (+ 5291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:37.449360 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:37.449374 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:37.449386 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:37.449693 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:37.450267 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:37.450275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:37.450281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:37.451113 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:37.451375 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:37.452300 (+ 925us) server_negotiation.cc:299] Negotiation successful
0619 16:59:37.452845 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":306,"threads_started":1}
I20260619 16:59:37.457507 368 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42589
I20260619 16:59:37.457918 368 heartbeater.cc:464] Registering TS with master...
I20260619 16:59:37.458781 368 heartbeater.cc:511] Master 127.16.48.62:42589 requested a full tablet report, sending...
I20260619 16:59:37.461680 32651 ts_manager.cc:194] Registered new tserver with Master: c1bc6ec497674e8cb3558d50ebf7b74f (127.16.48.1:36761)
I20260619 16:59:37.463948 32651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:49897
W20260619 16:59:37.784475 372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:59:37.784799 372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:59:37.784896 372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:59:37.792630 372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:59:37.792760 372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:59:37.792883 372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 16:59:37.803272 372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:43961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=36517
--tserver_master_addrs=127.16.48.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:59:37.804726 372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:59:37.806155 372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:59:37.821842 377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:37.822217 380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:37.822280 378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:59:37.823451 372 server_base.cc:1034] running on GCE node
I20260619 16:59:37.824251 372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:59:37.825343 372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:59:37.826560 372 hybrid_clock.cc:648] HybridClock initialized: now 1781888377826492 us; error 72 us; skew 500 ppm
I20260619 16:59:37.827031 372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:59:37 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888377, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:59:37.830565 372 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 16:59:37.832448 372 webserver.cc:460] Webserver started at http://127.16.48.2:36517/ using document root <none> and password file <none>
I20260619 16:59:37.833245 372 fs_manager.cc:359] Metadata directory not provided
I20260619 16:59:37.833403 372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:59:37.839385 372 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 16:59:37.842612 387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:37.844313 372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:59:37.844532 372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "831f0f39690d4031ae7dc05ce7938625"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "858ec586717582799a321bd633174c47"
server_key_iv: "f6d78fbdbb407f64e431b86bb007cfd1"
server_key_version: "encryptionkey@0"
I20260619 16:59:37.845738 372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:59:37.870685 372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:59:37.875366 372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:59:37.875687 372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:59:37.876814 372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:59:37.878334 372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:59:37.878494 372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:37.878626 372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:59:37.878715 372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:37.914857 372 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:43961
I20260619 16:59:37.914952 499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:43961 every 8 connection(s)
I20260619 16:59:37.916522 372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 16:59:37.924295 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 372
I20260619 16:59:37.925119 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:32841
--local_ip_for_outbound_sockets=127.16.48.3
--tserver_master_addrs=127.16.48.62:42589
--webserver_port=40761
--webserver_interface=127.16.48.3
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:59:37 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888377, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:59:37.940955 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:37.919399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:48677 (local address 127.16.48.62:42589)
0619 16:59:37.919659 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:37.919665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:37.921207 (+ 1542us) server_negotiation.cc:407] Connection header received
0619 16:59:37.923153 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:37.923158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:37.923281 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:37.923381 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:37.925832 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:37.926591 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:37.929332 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:37.929783 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:37.929970 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:37.933989 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:37.934021 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:37.934025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:37.934076 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:37.937101 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:37.938298 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:37.938302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:37.938305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:37.938479 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:37.939031 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:37.939034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:37.939036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:37.939344 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:37.939460 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:37.940483 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0619 16:59:37.940697 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 16:59:37.942692 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:37.919885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:48677)
0619 16:59:37.921002 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:37.921068 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:37.922847 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:37.923663 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:37.923673 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:37.924602 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:59:37.925593 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:37.925625 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:37.926798 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:37.926807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:37.929137 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:37.929157 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:37.930084 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:37.930092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:37.930361 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:37.931163 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:59:37.931189 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:59:37.933653 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:59:37.937337 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:37.937351 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:37.937375 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:37.938139 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:37.938605 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:37.938613 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:37.938620 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:37.938906 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:37.939481 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:59:37.939493 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:59:37.940020 (+ 527us) client_negotiation.cc:769] Sending connection context
0619 16:59:37.940397 (+ 377us) client_negotiation.cc:240] Negotiation successful
0619 16:59:37.941063 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":403,"threads_started":1}
I20260619 16:59:37.944540 500 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42589
I20260619 16:59:37.944913 500 heartbeater.cc:464] Registering TS with master...
I20260619 16:59:37.945791 500 heartbeater.cc:511] Master 127.16.48.62:42589 requested a full tablet report, sending...
I20260619 16:59:37.947499 32651 ts_manager.cc:194] Registered new tserver with Master: 831f0f39690d4031ae7dc05ce7938625 (127.16.48.2:43961)
I20260619 16:59:37.948444 32651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:48677
W20260619 16:59:38.293252 503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:59:38.293573 503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:59:38.293744 503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:59:38.302045 503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:59:38.302232 503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:59:38.302472 503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 16:59:38.313499 503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:32841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=40761
--tserver_master_addrs=127.16.48.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:59:38.314949 503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:59:38.316552 503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:59:38.332207 511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:38.332309 509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:59:38.332350 508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:59:38.333441 503 server_base.cc:1034] running on GCE node
I20260619 16:59:38.334146 503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:59:38.335387 503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:59:38.336606 503 hybrid_clock.cc:648] HybridClock initialized: now 1781888378336529 us; error 68 us; skew 500 ppm
I20260619 16:59:38.336962 503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:59:38 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888378, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:59:38.340770 503 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 16:59:38.342741 503 webserver.cc:460] Webserver started at http://127.16.48.3:40761/ using document root <none> and password file <none>
I20260619 16:59:38.343551 503 fs_manager.cc:359] Metadata directory not provided
I20260619 16:59:38.343724 503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:59:38.350675 503 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 16:59:38.354310 518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:38.356065 503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:59:38.356397 503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "f29b94a9f30949d88a5a3d88f48a537e"
format_stamp: "Formatted at 2026-06-19 16:59:15 on dist-test-slave-3ffd"
server_key: "b832add1e678ca46fa331f2f608b3ad6"
server_key_iv: "d5e9114f135429ceab7903d53474dd0c"
server_key_version: "encryptionkey@0"
I20260619 16:59:38.357511 503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:59:38.371730 503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:59:38.376766 503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:59:38.377079 503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:59:38.378285 503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:59:38.379822 503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:59:38.379925 503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:59:38.380100 503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:59:38.380230 503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 16:59:38.416139 503 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:32841
I20260619 16:59:38.416252 630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:32841 every 8 connection(s)
I20260619 16:59:38.417850 503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 16:59:38.425889 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 503
Jun 19 16:59:38 dist-test-slave-3ffd krb5kdc[32067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888378, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 16:59:38.443495 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.420834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:35449 (local address 127.16.48.62:42589)
0619 16:59:38.421019 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.421024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.422700 (+ 1676us) server_negotiation.cc:407] Connection header received
0619 16:59:38.424757 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.424763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.424865 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.424984 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:38.427239 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.428752 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.431499 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.432342 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.432483 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.436925 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:38.436930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:38.436933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:38.436969 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:38.439997 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:38.440874 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:38.440881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:38.440886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:38.441031 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:38.441553 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:38.441557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:38.441559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:38.441886 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:38.442059 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:38.442849 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 16:59:38.443098 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":61}
I20260619 16:59:38.444687 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.421537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:35449)
0619 16:59:38.422488 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:38.422542 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:38.424425 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:38.425261 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:38.425272 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:38.426164 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:59:38.426988 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.427015 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.429007 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.429013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.431312 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.431332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.432583 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.432589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.432876 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.433834 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:59:38.433866 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:59:38.436670 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:59:38.440204 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:38.440215 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:38.440236 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:38.440728 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:38.441170 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:59:38.441176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:59:38.441179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:59:38.441409 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:59:38.442038 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:59:38.442047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:59:38.442450 (+ 403us) client_negotiation.cc:769] Sending connection context
0619 16:59:38.442744 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 16:59:38.443371 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":386,"threads_started":1}
I20260619 16:59:38.445340 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.426981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.16.48.62:42589)
0619 16:59:38.428594 (+ 1613us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.428601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.429003 (+ 402us) server_negotiation.cc:407] Connection header received
0619 16:59:38.429156 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.429160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.429235 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.429321 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:38.431499 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.432738 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.434824 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.435267 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.435433 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.438894 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:38.438900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:38.438903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:38.438950 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:38.441840 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:38.442640 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:38.442646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:38.442649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:38.442781 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:38.443518 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:38.443522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:38.443524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:38.443945 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:38.444248 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:38.444647 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 16:59:38.444889 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1467,"thread_start_us":1335,"threads_started":1}
I20260619 16:59:38.446452 631 heartbeater.cc:347] Connected to a master server at 127.16.48.62:42589
I20260619 16:59:38.446852 631 heartbeater.cc:464] Registering TS with master...
I20260619 16:59:38.447647 631 heartbeater.cc:511] Master 127.16.48.62:42589 requested a full tablet report, sending...
I20260619 16:59:38.449345 32650 ts_manager.cc:194] Registered new tserver with Master: f29b94a9f30949d88a5a3d88f48a537e (127.16.48.3:32841)
I20260619 16:59:38.450337 32650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:35449
I20260619 16:59:38.461651 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:59:38.468384 368 heartbeater.cc:503] Master 127.16.48.62:42589 was elected leader, sending a full tablet report...
I20260619 16:59:38.469358 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.462971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.16.48.62:42589)
0619 16:59:38.463214 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.463222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.463627 (+ 405us) server_negotiation.cc:407] Connection header received
0619 16:59:38.463902 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.463906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.464061 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.464296 (+ 235us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:59:38.464824 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.465563 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.467143 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.467611 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.467762 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.468354 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:59:38.468619 (+ 265us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 16:59:38.469075 (+ 456us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52978: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 16:59:38.469523 635 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52978: token expired
W20260619 16:59:38.470132 32575 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.48.62:42589)
I20260619 16:59:38.484241 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.470670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.16.48.62:42589)
0619 16:59:38.470826 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.470831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.470870 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:59:38.471017 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.471021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.471091 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.471171 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:38.472745 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.473603 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.474993 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.475475 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.475596 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.478568 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:38.478574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:38.478578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:38.478618 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:38.481383 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:38.482190 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:38.482196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:38.482198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:38.482356 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:38.482810 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:38.482813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:38.482815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:38.483168 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:38.483359 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:38.483690 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 16:59:38.483888 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:59:38.488346 32650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:59:38.495801 32650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:59:38.523356 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.509196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:36761 (local address 127.0.0.1:41992)
0619 16:59:38.512207 (+ 3011us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:38.512253 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:38.512468 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:38.514361 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:38.514369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:38.514400 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:38.514996 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.515018 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.517276 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.517281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.520300 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.520313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.522181 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.522187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.522410 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.522464 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:59:38.522628 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 16:59:38.522855 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2701,"mutex_wait_us":129,"spinlock_wait_cycles":10880,"thread_start_us":3723,"threads_started":1}
I20260619 16:59:38.523641 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.509935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.16.48.2:43961)
0619 16:59:38.511487 (+ 1552us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.511501 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.513354 (+ 1853us) server_negotiation.cc:407] Connection header received
0619 16:59:38.513706 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.513717 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.514008 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.514190 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:38.515410 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.517106 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.520080 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.521654 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.521798 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.522025 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:38.523085 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0619 16:59:38.523299 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1311,"thread_start_us":1114,"threads_started":1}
I20260619 16:59:38.524606 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.509564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:43961 (local address 127.0.0.1:59716)
0619 16:59:38.513210 (+ 3646us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:38.513232 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:38.513476 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:38.514543 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:38.514550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:38.514598 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:38.515179 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.515201 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.517276 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.517281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.519891 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.519905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.522659 (+ 2754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.522666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.522852 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.522885 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:59:38.523945 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0619 16:59:38.524070 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3511,"mutex_wait_us":115,"spinlock_wait_cycles":1920,"thread_start_us":2184,"threads_started":1}
I20260619 16:59:38.524776 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.509400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.16.48.1:36761)
0619 16:59:38.513705 (+ 4305us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.513716 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.513747 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:59:38.513862 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.513869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.514120 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.514347 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:38.515345 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.517101 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.520494 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.521944 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.522097 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.522318 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:38.524215 (+ 1897us) server_negotiation.cc:299] Negotiation successful
0619 16:59:38.524420 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4118,"thread_start_us":3990,"threads_started":1}
I20260619 16:59:38.524953 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.513168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.16.48.3:32841)
0619 16:59:38.514044 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.514052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.516607 (+ 2555us) server_negotiation.cc:407] Connection header received
0619 16:59:38.516818 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.516824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.517047 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.517260 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:38.518292 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.519958 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.521732 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.523107 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.523239 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.523481 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:38.524337 (+ 856us) server_negotiation.cc:299] Negotiation successful
0619 16:59:38.524531 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":1428,"threads_started":1}
I20260619 16:59:38.525786 646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.514407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:32841 (local address 127.0.0.1:50626)
0619 16:59:38.516468 (+ 2061us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:38.516492 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:38.516644 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:38.517276 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:38.517281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:38.517308 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:38.518000 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.518014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.520161 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.520168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.521567 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.521580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.523626 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.523631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.523807 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.523841 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:59:38.523934 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:59:38.524095 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1928,"mutex_wait_us":4,"thread_start_us":1781,"threads_started":1}
I20260619 16:59:38.529103 303 tablet_service.cc:1467] Processing CreateTablet for tablet 78d054fda492420ca048ccb1b8f0e24e (DEFAULT_TABLE table=test-table [id=bf4068bbc0004c91bf509036eef836e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:59:38.529072 435 tablet_service.cc:1467] Processing CreateTablet for tablet 78d054fda492420ca048ccb1b8f0e24e (DEFAULT_TABLE table=test-table [id=bf4068bbc0004c91bf509036eef836e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:59:38.529735 566 tablet_service.cc:1467] Processing CreateTablet for tablet 78d054fda492420ca048ccb1b8f0e24e (DEFAULT_TABLE table=test-table [id=bf4068bbc0004c91bf509036eef836e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:59:38.530628 435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d054fda492420ca048ccb1b8f0e24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:59:38.530894 303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d054fda492420ca048ccb1b8f0e24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:59:38.531200 566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d054fda492420ca048ccb1b8f0e24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:59:38.544633 647 tablet_bootstrap.cc:492] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: Bootstrap starting.
I20260619 16:59:38.547724 648 tablet_bootstrap.cc:492] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: Bootstrap starting.
I20260619 16:59:38.548779 647 tablet_bootstrap.cc:654] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: Neither blocks nor log segments found. Creating new log.
I20260619 16:59:38.550406 647 log.cc:824] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: Log is configured to *not* fsync() on all Append() calls
I20260619 16:59:38.551807 648 tablet_bootstrap.cc:654] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: Neither blocks nor log segments found. Creating new log.
I20260619 16:59:38.553251 648 log.cc:824] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:59:38.554805 647 tablet_bootstrap.cc:492] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: No bootstrap required, opened a new log
I20260619 16:59:38.555186 647 ts_tablet_manager.cc:1397] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 16:59:38.556968 649 tablet_bootstrap.cc:492] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: Bootstrap starting.
I20260619 16:59:38.557255 648 tablet_bootstrap.cc:492] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: No bootstrap required, opened a new log
I20260619 16:59:38.557633 648 ts_tablet_manager.cc:1397] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260619 16:59:38.560352 649 tablet_bootstrap.cc:654] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: Neither blocks nor log segments found. Creating new log.
I20260619 16:59:38.561609 649 log.cc:824] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:59:38.565347 649 tablet_bootstrap.cc:492] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: No bootstrap required, opened a new log
I20260619 16:59:38.565696 649 ts_tablet_manager.cc:1397] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260619 16:59:38.567890 647 raft_consensus.cc:348] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } }
I20260619 16:59:38.568468 647 raft_consensus.cc:374] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:59:38.568586 647 raft_consensus.cc:729] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831f0f39690d4031ae7dc05ce7938625, State: Initialized, Role: FOLLOWER
I20260619 16:59:38.568964 648 raft_consensus.cc:348] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } }
I20260619 16:59:38.569460 648 raft_consensus.cc:374] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:59:38.569310 647 consensus_queue.cc:260] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } }
I20260619 16:59:38.569581 648 raft_consensus.cc:729] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1bc6ec497674e8cb3558d50ebf7b74f, State: Initialized, Role: FOLLOWER
I20260619 16:59:38.570302 648 consensus_queue.cc:260] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } }
I20260619 16:59:38.572880 648 ts_tablet_manager.cc:1428] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260619 16:59:38.574870 649 raft_consensus.cc:348] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } }
I20260619 16:59:38.575316 649 raft_consensus.cc:374] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:59:38.575439 649 raft_consensus.cc:729] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29b94a9f30949d88a5a3d88f48a537e, State: Initialized, Role: FOLLOWER
I20260619 16:59:38.576313 500 heartbeater.cc:503] Master 127.16.48.62:42589 was elected leader, sending a full tablet report...
I20260619 16:59:38.576325 649 consensus_queue.cc:260] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } }
I20260619 16:59:38.579022 631 heartbeater.cc:503] Master 127.16.48.62:42589 was elected leader, sending a full tablet report...
I20260619 16:59:38.579896 649 ts_tablet_manager.cc:1428] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 16:59:38.580744 647 ts_tablet_manager.cc:1428] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: Time spent starting tablet: real 0.025s user 0.022s sys 0.000s
W20260619 16:59:38.668680 501 tablet.cc:2369] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:59:38.669670 632 tablet.cc:2369] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:59:38.672883 369 tablet.cc:2369] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:59:38.913686 653 raft_consensus.cc:482] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:59:38.914073 653 raft_consensus.cc:504] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } }
I20260619 16:59:38.916225 653 leader_election.cc:288] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1bc6ec497674e8cb3558d50ebf7b74f (127.16.48.1:36761), f29b94a9f30949d88a5a3d88f48a537e (127.16.48.3:32841)
I20260619 16:59:38.926118 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.916603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:53929 (local address 127.16.48.1:36761)
0619 16:59:38.916840 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.916846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.918085 (+ 1239us) server_negotiation.cc:407] Connection header received
0619 16:59:38.918249 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.918254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.918332 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.918427 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:38.919286 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.920442 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.924078 (+ 3636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.925008 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.925232 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.925359 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:38.925701 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:59:38.925854 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:59:38.926486 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.916931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:56609 (local address 127.16.48.3:32841)
0619 16:59:38.917130 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.917135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.917893 (+ 758us) server_negotiation.cc:407] Connection header received
0619 16:59:38.918244 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.918251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.918332 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.918401 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:59:38.919240 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.920431 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.923862 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.925192 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.925337 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.925446 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:38.926056 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 16:59:38.926222 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:59:38.926493 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.917210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:36761 (local address 127.16.48.2:53929)
0619 16:59:38.917939 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:38.917958 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:38.918098 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:38.918475 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:38.918479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:38.918510 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:38.919035 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.919055 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.920625 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.920642 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.923603 (+ 2961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.923906 (+ 303us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.925303 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.925309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.925494 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.925521 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:59:38.925666 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:59:38.925831 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":178,"thread_start_us":412,"threads_started":1}
I20260619 16:59:38.927222 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.916470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:32841 (local address 127.16.48.2:56609)
0619 16:59:38.917743 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0619 16:59:38.917767 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:59:38.918047 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:59:38.918644 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:59:38.918647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:59:38.918680 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:59:38.919070 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.919078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.920625 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.920630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.923603 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:59:38.923628 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.925634 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.925641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:59:38.925843 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.925877 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:59:38.926676 (+ 799us) client_negotiation.cc:240] Negotiation successful
0619 16:59:38.926836 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":206,"thread_start_us":816,"threads_started":1}
I20260619 16:59:38.928000 323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d054fda492420ca048ccb1b8f0e24e" candidate_uuid: "831f0f39690d4031ae7dc05ce7938625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" is_pre_election: true
I20260619 16:59:38.928236 586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d054fda492420ca048ccb1b8f0e24e" candidate_uuid: "831f0f39690d4031ae7dc05ce7938625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29b94a9f30949d88a5a3d88f48a537e" is_pre_election: true
I20260619 16:59:38.928704 586 raft_consensus.cc:2454] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831f0f39690d4031ae7dc05ce7938625 in term 0.
I20260619 16:59:38.928704 323 raft_consensus.cc:2454] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831f0f39690d4031ae7dc05ce7938625 in term 0.
I20260619 16:59:38.929452 388 leader_election.cc:302] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831f0f39690d4031ae7dc05ce7938625, c1bc6ec497674e8cb3558d50ebf7b74f; no voters:
I20260619 16:59:38.929970 653 raft_consensus.cc:2781] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:59:38.930143 653 raft_consensus.cc:482] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:59:38.930245 653 raft_consensus.cc:3037] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:59:38.932709 653 raft_consensus.cc:504] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } }
I20260619 16:59:38.933369 653 leader_election.cc:288] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [CANDIDATE]: Term 1 election: Requested vote from peers c1bc6ec497674e8cb3558d50ebf7b74f (127.16.48.1:36761), f29b94a9f30949d88a5a3d88f48a537e (127.16.48.3:32841)
I20260619 16:59:38.934046 586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d054fda492420ca048ccb1b8f0e24e" candidate_uuid: "831f0f39690d4031ae7dc05ce7938625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29b94a9f30949d88a5a3d88f48a537e"
I20260619 16:59:38.934176 323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d054fda492420ca048ccb1b8f0e24e" candidate_uuid: "831f0f39690d4031ae7dc05ce7938625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f"
I20260619 16:59:38.934312 586 raft_consensus.cc:3037] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:59:38.934401 323 raft_consensus.cc:3037] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:59:38.937829 323 raft_consensus.cc:2454] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831f0f39690d4031ae7dc05ce7938625 in term 1.
I20260619 16:59:38.937829 586 raft_consensus.cc:2454] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831f0f39690d4031ae7dc05ce7938625 in term 1.
I20260619 16:59:38.938443 390 leader_election.cc:302] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831f0f39690d4031ae7dc05ce7938625, f29b94a9f30949d88a5a3d88f48a537e; no voters:
I20260619 16:59:38.938882 653 raft_consensus.cc:2781] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:59:38.939407 653 raft_consensus.cc:686] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 1 LEADER]: Becoming Leader. State: Replica: 831f0f39690d4031ae7dc05ce7938625, State: Running, Role: LEADER
I20260619 16:59:38.940060 653 consensus_queue.cc:237] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } }
I20260619 16:59:38.944969 32650 catalog_manager.cc:5495] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 831f0f39690d4031ae7dc05ce7938625 (127.16.48.2). New cstate: current_term: 1 leader_uuid: "831f0f39690d4031ae7dc05ce7938625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831f0f39690d4031ae7dc05ce7938625" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 } health_report { overall_health: UNKNOWN } } }
I20260619 16:59:38.983572 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:38.977887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.16.48.2:43961)
0619 16:59:38.978177 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:38.978182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:38.978202 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:59:38.978383 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:38.978387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:38.978469 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:38.978617 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:59:38.979047 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.979785 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.981375 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:38.981775 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:38.981872 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:38.982400 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:59:38.982934 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:59:38.983043 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:38.983167 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 16:59:38.983241 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 16:59:38.992576 586 raft_consensus.cc:1261] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [term 1 FOLLOWER]: Refusing update from remote peer 831f0f39690d4031ae7dc05ce7938625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:59:38.992599 323 raft_consensus.cc:1261] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [term 1 FOLLOWER]: Refusing update from remote peer 831f0f39690d4031ae7dc05ce7938625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:59:38.993685 658 consensus_queue.cc:1035] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b94a9f30949d88a5a3d88f48a537e" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:59:38.994172 653 consensus_queue.cc:1035] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1bc6ec497674e8cb3558d50ebf7b74f" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:59:39.004846 661 mvcc.cc:205] Tried to move back new op lower bound from 7298614800342630400 to 7298614800142589952. Current Snapshot: MvccSnapshot[applied={T|T < 7298614800342630400}]
I20260619 16:59:39.006366 662 mvcc.cc:205] Tried to move back new op lower bound from 7298614800342630400 to 7298614800142589952. Current Snapshot: MvccSnapshot[applied={T|T < 7298614800342630400}]
I20260619 16:59:39.023339 32651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:59:39.023644 32651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:59:39.027666 32651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Sending DeleteTablet for 3 replicas of tablet 78d054fda492420ca048ccb1b8f0e24e
I20260619 16:59:39.029052 303 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d054fda492420ca048ccb1b8f0e24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:59:39 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:41992
I20260619 16:59:39.029174 435 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d054fda492420ca048ccb1b8f0e24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:59:39 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:59716
I20260619 16:59:39.029793 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 32688
I20260619 16:59:39.030385 566 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d054fda492420ca048ccb1b8f0e24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:59:39 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:50626
I20260619 16:59:39.045164 673 tablet_replica.cc:330] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: stopping tablet replica
I20260619 16:59:39.045867 673 raft_consensus.cc:2229] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:59:39.046581 673 raft_consensus.cc:2258] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:59:39.048612 672 tablet_replica.cc:330] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: stopping tablet replica
I20260619 16:59:39.049388 672 raft_consensus.cc:2229] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:59:39.050817 672 raft_consensus.cc:2258] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:59:39.052667 673 ts_tablet_manager.cc:1905] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:59:39.053194 672 ts_tablet_manager.cc:1905] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:59:39.061538 672 ts_tablet_manager.cc:1918] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:59:39.061914 672 log.cc:1196] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/78d054fda492420ca048ccb1b8f0e24e
I20260619 16:59:39.062731 672 ts_tablet_manager.cc:1939] T 78d054fda492420ca048ccb1b8f0e24e P 831f0f39690d4031ae7dc05ce7938625: Deleting consensus metadata
I20260619 16:59:39.064356 671 tablet_replica.cc:330] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: stopping tablet replica
I20260619 16:59:39.064873 671 raft_consensus.cc:2229] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:59:39.065181 32637 catalog_manager.cc:4841] TS 831f0f39690d4031ae7dc05ce7938625 (127.16.48.2:43961): tablet 78d054fda492420ca048ccb1b8f0e24e (table test-table [id=bf4068bbc0004c91bf509036eef836e6]) successfully deleted
I20260619 16:59:39.065492 671 raft_consensus.cc:2258] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:59:39.067701 671 ts_tablet_manager.cc:1905] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:59:39.070461 673 ts_tablet_manager.cc:1918] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:59:39.070751 673 log.cc:1196] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/78d054fda492420ca048ccb1b8f0e24e
I20260619 16:59:39.071416 673 ts_tablet_manager.cc:1939] T 78d054fda492420ca048ccb1b8f0e24e P f29b94a9f30949d88a5a3d88f48a537e: Deleting consensus metadata
I20260619 16:59:39.073982 32637 catalog_manager.cc:4841] TS f29b94a9f30949d88a5a3d88f48a537e (127.16.48.3:32841): tablet 78d054fda492420ca048ccb1b8f0e24e (table test-table [id=bf4068bbc0004c91bf509036eef836e6]) successfully deleted
I20260619 16:59:39.075891 671 ts_tablet_manager.cc:1918] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:59:39.076097 671 log.cc:1196] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/78d054fda492420ca048ccb1b8f0e24e
I20260619 16:59:39.076694 671 ts_tablet_manager.cc:1939] T 78d054fda492420ca048ccb1b8f0e24e P c1bc6ec497674e8cb3558d50ebf7b74f: Deleting consensus metadata
I20260619 16:59:39.078848 32635 catalog_manager.cc:4841] TS c1bc6ec497674e8cb3558d50ebf7b74f (127.16.48.1:36761): tablet 78d054fda492420ca048ccb1b8f0e24e (table test-table [id=bf4068bbc0004c91bf509036eef836e6]) successfully deleted
I20260619 16:59:39.079116 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:39.031386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.16.48.1:36761)
0619 16:59:39.031642 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:39.031652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:39.040850 (+ 9198us) server_negotiation.cc:407] Connection header received
0619 16:59:39.041214 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:39.041224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:39.041357 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:39.041457 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:39.047187 (+ 5730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:39.048896 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:39.062387 (+ 13491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:39.063601 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:39.063798 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:39.069782 (+ 5984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:39.069801 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:39.069821 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:39.069878 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:39.074392 (+ 4514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:39.075532 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:39.075546 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:39.075552 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:39.075748 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:39.076335 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:39.076340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:39.076343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:39.076887 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:39.077205 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:39.077933 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 16:59:39.078182 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:59:39.080711 363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:39.175110 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 32688
I20260619 16:59:39.194509 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 372
I20260619 16:59:39.213471 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:39.195440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.16.48.2:43961)
0619 16:59:39.195648 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:39.195655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:39.195746 (+ 91us) server_negotiation.cc:407] Connection header received
0619 16:59:39.195905 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:39.195912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:39.196038 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:39.196219 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:39.197863 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:39.198938 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:39.201411 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:39.201932 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:39.202081 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:39.205838 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:39.205857 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:39.205875 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:39.205926 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:39.209741 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:39.210845 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:39.210854 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:39.210856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:39.211054 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:39.211603 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:39.211608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:39.211610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:39.212095 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:39.212533 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:39.212896 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:59:39.213103 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 16:59:39.214404 495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:39.332072 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 372
I20260619 16:59:39.352281 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 503
I20260619 16:59:39.367908 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:59:39.353107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.16.48.3:32841)
0619 16:59:39.353323 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:59:39.353329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:59:39.353351 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:59:39.353574 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:59:39.353587 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:59:39.353681 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:59:39.353795 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:59:39.355146 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:39.355905 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:39.357518 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:59:39.357949 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:59:39.358107 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:59:39.361284 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:59:39.361299 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:59:39.361315 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:59:39.361358 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:59:39.364740 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:39.365696 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:39.365705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:39.365707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:39.365878 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:59:39.366432 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:59:39.366436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:59:39.366438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:59:39.366863 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:59:39.367144 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 16:59:39.367434 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 16:59:39.367633 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:59:39.368701 626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:39.459519 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 503
I20260619 16:59:39.477260 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 32618
I20260619 16:59:39.478590 32681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:59:39.595428 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 32618
I20260619 16:59:59.618968 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42589
--webserver_interface=127.16.48.62
--webserver_port=32923
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:42589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:59:59.975099 679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:59:59.975430 679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:59:59.975607 679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:59:59.983845 679 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:59:59.983963 679 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:59:59.984050 679 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:59:59.984166 679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:59:59.984277 679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:59:59.984387 679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:59:59.984477 679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:59:59.984560 679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:59:59.995375 679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:42589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=32923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:59:59.997238 679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:59:59.999239 679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:00.012717 684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:00.012989 685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:00.013614 679 server_base.cc:1034] running on GCE node
W20260619 17:00:00.014155 687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:00.015194 679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:00.016997 679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:00.018303 679 hybrid_clock.cc:648] HybridClock initialized: now 1781888400018239 us; error 41 us; skew 500 ppm
I20260619 17:00:00.018707 679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:00 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888400, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:00.031339 679 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:00:00.033263 679 webserver.cc:460] Webserver started at http://127.16.48.62:32923/ using document root <none> and password file <none>
I20260619 17:00:00.034070 679 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:00.034175 679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:00.040781 679 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 17:00:00.045837 694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:00.047572 679 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 17:00:00.047827 679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c"
format_stamp: "Formatted at 2026-06-19 16:59:13 on dist-test-slave-3ffd"
server_key: "835bdcfa1aaf68fd66ea0b3641dca61f"
server_key_iv: "026370113ea06e7a0e25e60ffb30f30a"
server_key_version: "encryptionkey@0"
I20260619 17:00:00.049007 679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:00.069260 679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:00.073791 679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:00.074136 679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:00.094810 679 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:42589
I20260619 17:00:00.094842 745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:42589 every 8 connection(s)
I20260619 17:00:00.096593 679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:00.104034 746 sys_catalog.cc:263] Verifying existing consensus state
I20260619 17:00:00.106215 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 679
I20260619 17:00:00.107090 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:36761
--local_ip_for_outbound_sockets=127.16.48.1
--tserver_master_addrs=127.16.48.62:42589
--webserver_port=38169
--webserver_interface=127.16.48.1
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 17:00:00.108018 746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Bootstrap starting.
I20260619 17:00:00.130403 746 log.cc:824] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:00.147768 746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 17:00:00.148480 746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Bootstrap complete.
I20260619 17:00:00.158519 746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 17:00:00.159569 746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6119b6ac2a4a6dbad4fd57fc35ba5c, State: Initialized, Role: FOLLOWER
I20260619 17:00:00.160319 746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 17:00:00.160586 746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:00.160722 746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:00.160908 746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 2 FOLLOWER]: Advancing to term 3
I20260619 17:00:00.165181 746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 17:00:00.165773 746 leader_election.cc:302] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c6119b6ac2a4a6dbad4fd57fc35ba5c; no voters:
I20260619 17:00:00.166519 746 leader_election.cc:288] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 17:00:00.166676 750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 3 FOLLOWER]: Leader election won for term 3
I20260619 17:00:00.167964 750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [term 3 LEADER]: Becoming Leader. State: Replica: 6c6119b6ac2a4a6dbad4fd57fc35ba5c, State: Running, Role: LEADER
I20260619 17:00:00.168697 750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } }
I20260619 17:00:00.169052 746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:00.171451 752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c6119b6ac2a4a6dbad4fd57fc35ba5c. Latest consensus state: current_term: 3 leader_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } } }
I20260619 17:00:00.171857 752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:00.172523 751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6119b6ac2a4a6dbad4fd57fc35ba5c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42589 } } }
I20260619 17:00:00.172834 751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:00.183669 756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:00.194053 756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:00.195843 756 catalog_manager.cc:1259] Loaded cluster ID: edc6c2f572324e93828489e0bdc92058
I20260619 17:00:00.196051 756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:00.200855 756 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:00.202983 756 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Loaded TSK: 0
I20260619 17:00:00.203295 756 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c6119b6ac2a4a6dbad4fd57fc35ba5c: Loaded TSK: 1
F20260619 17:00:00.205000 756 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f35ecaf59ad google::LogMessage::Fail() at ??:0
@ 0x7f35ecaf888a google::LogMessage::SendToLog() at ??:0
@ 0x7f35ecaf54de google::LogMessage::Flush() at ??:0
@ 0x7f35ecaf6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f35f08597d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f35f084873e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260619 17:00:00.223537 768 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f36025241c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3602523783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f36025af2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f36025af0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f36025f95e9 std::function<>::operator()() at ??:0
@ 0x7f3602527cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3602525c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f36025ad610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f36025ad439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f35ef741345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f35ef74491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f35ef744799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f35ef71feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f35f29f96db start_thread at ??:0
@ 0x7f35eb98f61f clone at ??:0
W20260619 17:00:00.546083 748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:00.546388 748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:00.546546 748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:00.554265 748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:00.554397 748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:00.554605 748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 17:00:00.564999 748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:36761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=38169
--tserver_master_addrs=127.16.48.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:00.566426 748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:00.567859 748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:00.582870 796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:00.582922 794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:00.582821 793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:00.585868 748 server_base.cc:1034] running on GCE node
I20260619 17:00:00.586694 748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:00.587846 748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:00.589087 748 hybrid_clock.cc:648] HybridClock initialized: now 1781888400589022 us; error 56 us; skew 500 ppm
I20260619 17:00:00.589483 748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:00 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888400, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:00.593129 748 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 17:00:00.594928 748 webserver.cc:460] Webserver started at http://127.16.48.1:38169/ using document root <none> and password file <none>
I20260619 17:00:00.595685 748 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:00.595796 748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:00.601691 748 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 17:00:00.604777 803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:00.606295 748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 17:00:00.606496 748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "c1bc6ec497674e8cb3558d50ebf7b74f"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "0881694c21e7145406f9f6a1ef61b009"
server_key_iv: "e6a4731943fb971372ea1cd1f22fba9f"
server_key_version: "encryptionkey@0"
I20260619 17:00:00.607569 748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:00.621202 748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:00.625746 748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:00.626056 748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:00.627152 748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:00.628690 748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:00.628813 748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:00.628955 748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:00.629071 748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:00.664938 748 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:36761
I20260619 17:00:00.665071 915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:36761 every 8 connection(s)
I20260619 17:00:00.666553 748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 17:00:00.671967 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:00.669909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:52651)
0619 17:00:00.670738 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:00.670972 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":328,"threads_started":1}
W20260619 17:00:00.672907 916 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:42589 (0 consecutive failures): Network error: Failed to ping master at 127.16.48.62:42589: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
I20260619 17:00:00.673985 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:00.673547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:38793)
0619 17:00:00.673750 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:00.673834 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 17:00:00.674746 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 748
I20260619 17:00:00.675204 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:00.674729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:55797)
0619 17:00:00.674928 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:00.675031 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 17:00:00.675545 916 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 17:00:00.675561 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:43961
--local_ip_for_outbound_sockets=127.16.48.2
--tserver_master_addrs=127.16.48.62:42589
--webserver_port=36517
--webserver_interface=127.16.48.2
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:01.033211 919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:01.033531 919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:01.033704 919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:01.041800 919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:01.041951 919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:01.042171 919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 17:00:01.053001 919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:43961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=36517
--tserver_master_addrs=127.16.48.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:01.054437 919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:01.055855 919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:01.070430 927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:01.070379 925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:01.070400 924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:01.071702 919 server_base.cc:1034] running on GCE node
I20260619 17:00:01.072469 919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:01.073498 919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:01.074715 919 hybrid_clock.cc:648] HybridClock initialized: now 1781888401074650 us; error 59 us; skew 500 ppm
I20260619 17:00:01.075044 919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:01 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:01.078424 919 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 17:00:01.080044 919 webserver.cc:460] Webserver started at http://127.16.48.2:36517/ using document root <none> and password file <none>
I20260619 17:00:01.080783 919 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:01.080888 919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:01.087044 919 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 17:00:01.090243 934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:01.091897 919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 17:00:01.092079 919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "831f0f39690d4031ae7dc05ce7938625"
format_stamp: "Formatted at 2026-06-19 16:59:14 on dist-test-slave-3ffd"
server_key: "858ec586717582799a321bd633174c47"
server_key_iv: "f6d78fbdbb407f64e431b86bb007cfd1"
server_key_version: "encryptionkey@0"
I20260619 17:00:01.093205 919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:01.119352 919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:01.124401 919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:01.124660 919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:01.125767 919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:01.127342 919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:01.127442 919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:01.127523 919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:01.127581 919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:01.166132 919 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:43961
I20260619 17:00:01.166199 1046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:43961 every 8 connection(s)
I20260619 17:00:01.167835 919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 17:00:01.172695 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 919
I20260619 17:00:01.173564 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:32841
--local_ip_for_outbound_sockets=127.16.48.3
--tserver_master_addrs=127.16.48.62:42589
--webserver_port=40761
--webserver_interface=127.16.48.3
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 17:00:01.174644 1049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:01.171311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:45579)
0619 17:00:01.172767 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:01.173156 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":521,"threads_started":1}
W20260619 17:00:01.175669 1047 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:42589 (0 consecutive failures): Network error: Failed to ping master at 127.16.48.62:42589: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
I20260619 17:00:01.176970 1049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:01.176443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:54905)
0619 17:00:01.176666 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:01.176778 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 17:00:01.178397 1049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:01.177965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:39319)
0619 17:00:01.178148 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:01.178257 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 17:00:01.178779 1047 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 17:00:01.544180 1050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:01.544523 1050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:01.544725 1050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:01.552736 1050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:01.552881 1050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:01.553066 1050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 17:00:01.563494 1050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:32841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=40761
--tserver_master_addrs=127.16.48.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:01.565052 1050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:01.566541 1050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:01.583557 1055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:01.584446 1058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:01.584535 1056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:01.586118 1050 server_base.cc:1034] running on GCE node
I20260619 17:00:01.587042 1050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:01.588304 1050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:01.589890 1050 hybrid_clock.cc:648] HybridClock initialized: now 1781888401589654 us; error 41 us; skew 500 ppm
I20260619 17:00:01.590493 1050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:01 dist-test-slave-3ffd krb5kdc[32067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:01.595113 1050 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 17:00:01.597586 1050 webserver.cc:460] Webserver started at http://127.16.48.3:40761/ using document root <none> and password file <none>
I20260619 17:00:01.598383 1050 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:01.598522 1050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:01.605809 1050 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 17:00:01.609531 1065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:01.611469 1050 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 17:00:01.611716 1050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "f29b94a9f30949d88a5a3d88f48a537e"
format_stamp: "Formatted at 2026-06-19 16:59:15 on dist-test-slave-3ffd"
server_key: "b832add1e678ca46fa331f2f608b3ad6"
server_key_iv: "d5e9114f135429ceab7903d53474dd0c"
server_key_version: "encryptionkey@0"
I20260619 17:00:01.612998 1050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:01.641944 1050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:01.647966 1050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:01.648380 1050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:01.649807 1050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:01.651865 1050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:01.652033 1050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:01.652211 1050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:01.652307 1050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:01.677417 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:01.676371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:38415)
0619 17:00:01.676939 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:01.677071 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":277,"threads_started":1}
I20260619 17:00:01.693316 1050 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:32841
I20260619 17:00:01.693434 1178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:32841 every 8 connection(s)
I20260619 17:00:01.694950 1050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 17:00:01.701350 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:01.698971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:34573)
0619 17:00:01.700012 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:01.700376 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":488,"threads_started":1}
W20260619 17:00:01.702179 1179 heartbeater.cc:644] Failed to heartbeat to 127.16.48.62:42589 (0 consecutive failures): Network error: Failed to ping master at 127.16.48.62:42589: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
I20260619 17:00:01.703244 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:01.702825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:56643)
0619 17:00:01.703013 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:01.703116 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 17:00:01.704473 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:01.704059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:42983)
0619 17:00:01.704269 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:01.704350 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 17:00:01.704607 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 1050
W20260619 17:00:01.704735 1179 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 17:00:02.180437 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:02.179432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:48313)
0619 17:00:02.179987 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:02.180087 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":273,"threads_started":1}
I20260619 17:00:02.679690 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:02.678610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:36967)
0619 17:00:02.679148 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:02.679420 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":276,"threads_started":1}
I20260619 17:00:02.706671 1185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:02.705469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:45849)
0619 17:00:02.706159 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:02.706292 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":329,"threads_started":1}
I20260619 17:00:03.182540 1186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:03.181527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:49745)
0619 17:00:03.182226 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:03.182339 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":258,"threads_started":1}
I20260619 17:00:03.681900 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:03.681082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:34563)
0619 17:00:03.681600 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:03.681698 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":260,"threads_started":1}
I20260619 17:00:03.708653 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:03.707796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:36489)
0619 17:00:03.708399 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:03.708484 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":352,"threads_started":1}
I20260619 17:00:04.184770 1189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:04.183675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:53053)
0619 17:00:04.184318 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:04.184458 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":273,"threads_started":1}
I20260619 17:00:04.684092 1190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:04.683056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:53323)
0619 17:00:04.683742 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:04.683850 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":247,"threads_started":1}
I20260619 17:00:04.710640 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:04.709761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:52721)
0619 17:00:04.710307 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:04.710423 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":294,"threads_started":1}
I20260619 17:00:05.186854 1192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:05.185959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:49363)
0619 17:00:05.186550 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:05.186667 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":293,"threads_started":1}
I20260619 17:00:05.686262 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:05.685379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:37059)
0619 17:00:05.685951 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:05.686042 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":294,"threads_started":1}
I20260619 17:00:05.712744 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:05.711873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:60983)
0619 17:00:05.712473 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:05.712569 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":340,"threads_started":1}
I20260619 17:00:06.188786 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:06.187953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:40709)
0619 17:00:06.188517 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:06.188604 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":321,"threads_started":1}
I20260619 17:00:06.688067 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:06.687263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:51085)
0619 17:00:06.687799 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:06.687894 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":276,"threads_started":1}
I20260619 17:00:06.714576 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:06.713778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:47205)
0619 17:00:06.714285 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:06.714380 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":250,"threads_started":1}
I20260619 17:00:07.190584 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:07.189783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:42271)
0619 17:00:07.190277 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:07.190379 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
I20260619 17:00:07.690145 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:07.689234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:48617)
0619 17:00:07.689793 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:07.689885 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":264,"threads_started":1}
I20260619 17:00:07.716708 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:07.715722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:46869)
0619 17:00:07.716245 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:07.716344 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":239,"threads_started":1}
I20260619 17:00:08.192811 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:08.191715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:37817)
0619 17:00:08.192502 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:08.192593 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":478,"threads_started":1}
I20260619 17:00:08.692091 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:08.691296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:36105)
0619 17:00:08.691816 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:08.691904 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":279,"threads_started":1}
I20260619 17:00:08.718878 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:08.717843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:40293)
0619 17:00:08.718509 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:08.718607 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":341,"threads_started":1}
I20260619 17:00:09.194648 1204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:09.193854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:35293)
0619 17:00:09.194356 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:09.194446 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260619 17:00:09.694221 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:09.693292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:55249)
0619 17:00:09.693956 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:09.694050 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"threads_started":1}
I20260619 17:00:09.721210 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:09.720461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:34287)
0619 17:00:09.720955 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:09.721046 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":258,"threads_started":1}
I20260619 17:00:10.196667 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:10.195770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:50403)
0619 17:00:10.196346 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:10.196435 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":292,"threads_started":1}
I20260619 17:00:10.696653 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:10.695635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:36251)
0619 17:00:10.696307 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:10.696435 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":336,"threads_started":1}
I20260619 17:00:10.723095 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:10.722190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:54857)
0619 17:00:10.722822 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:10.722917 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":363,"threads_started":1}
I20260619 17:00:11.198694 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:11.197791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:43087)
0619 17:00:11.198350 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:11.198446 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":333,"threads_started":1}
I20260619 17:00:11.698742 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:11.697879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:50531)
0619 17:00:11.698453 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:11.698546 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":269,"threads_started":1}
I20260619 17:00:11.725126 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:11.724327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:51197)
0619 17:00:11.724854 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:11.724945 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":286,"threads_started":1}
I20260619 17:00:12.200762 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:12.199874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:57209)
0619 17:00:12.200455 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:12.200565 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":355,"threads_started":1}
I20260619 17:00:12.701014 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:12.699842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:44697)
0619 17:00:12.700487 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:12.700733 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":339,"threads_started":1}
I20260619 17:00:12.727108 1215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:12.726276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:48713)
0619 17:00:12.726811 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:12.726918 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":298,"threads_started":1}
I20260619 17:00:13.202742 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:13.201949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:35269)
0619 17:00:13.202456 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:13.202547 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":244,"threads_started":1}
I20260619 17:00:13.702837 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:13.702037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:51433)
0619 17:00:13.702518 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:13.702653 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
I20260619 17:00:13.729135 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:13.728317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:35033)
0619 17:00:13.728877 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:13.728971 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":313,"threads_started":1}
I20260619 17:00:14.205120 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:14.203950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:33551)
0619 17:00:14.204643 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:14.204762 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":345,"threads_started":1}
I20260619 17:00:14.704913 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:14.704076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:38175)
0619 17:00:14.704633 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:14.704723 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":303,"threads_started":1}
I20260619 17:00:14.730959 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:14.730069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:40819)
0619 17:00:14.730634 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:14.730745 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":245,"threads_started":1}
I20260619 17:00:15.207022 1222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:15.206278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:42151)
0619 17:00:15.206735 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:15.206827 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":212,"threads_started":1}
I20260619 17:00:15.706905 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:15.706124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:46535)
0619 17:00:15.706609 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:15.706709 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":236,"threads_started":1}
I20260619 17:00:15.732863 1224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:15.732018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:54631)
0619 17:00:15.732579 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:15.732672 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":314,"threads_started":1}
I20260619 17:00:16.209226 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:16.208410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.2:57987)
0619 17:00:16.208887 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:16.209010 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":227,"threads_started":1}
I20260619 17:00:16.709143 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:16.708185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.1:55831)
0619 17:00:16.708767 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:16.708897 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":312,"threads_started":1}
I20260619 17:00:16.716058 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 748
I20260619 17:00:16.734910 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:16.734011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:42589 (local address 127.16.48.3:56701)
0619 17:00:16.734521 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:16.734663 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.62:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":272,"threads_started":1}
I20260619 17:00:16.738302 1227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:16.717348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.16.48.1:36761)
0619 17:00:16.719805 (+ 2457us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:16.719820 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:16.719868 (+ 48us) server_negotiation.cc:407] Connection header received
0619 17:00:16.721662 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:16.721672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:16.722148 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:16.722475 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:16.724064 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:16.725495 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:16.726891 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:16.727611 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:16.727757 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:16.731161 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:16.731171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:16.731190 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:16.731245 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:16.734521 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:16.735525 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:16.735533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:16.735541 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:16.735746 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:16.736206 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:16.736210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:16.736212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:16.736848 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:16.737102 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:16.737625 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 17:00:16.738004 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2264,"thread_start_us":2071,"threads_started":1}
I20260619 17:00:16.739838 911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:16.826431 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 748
I20260619 17:00:16.845079 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 919
I20260619 17:00:16.864650 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:16.846021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.16.48.2:43961)
0619 17:00:16.846558 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:16.846574 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:16.846639 (+ 65us) server_negotiation.cc:407] Connection header received
0619 17:00:16.848313 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:16.848326 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:16.848787 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:16.849120 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:16.850536 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:16.851861 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:16.853477 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:16.854196 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:16.854355 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:16.857665 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:16.857673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:16.857691 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:16.857737 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:16.861020 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:16.861880 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:16.861887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:16.861894 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:16.862091 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:16.862447 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:16.862450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:16.862452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:16.863026 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:16.863272 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:16.863772 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 17:00:16.864327 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":247,"threads_started":1}
I20260619 17:00:16.866222 1042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:16.952593 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 919
I20260619 17:00:16.972173 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 1050
I20260619 17:00:16.991437 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:16.973313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.16.48.3:32841)
0619 17:00:16.973881 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:16.973896 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:16.973940 (+ 44us) server_negotiation.cc:407] Connection header received
0619 17:00:16.975765 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:16.975776 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:16.976307 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:16.976629 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:16.977942 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:16.979202 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:16.980606 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:16.981313 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:16.981463 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:16.984191 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:16.984207 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:16.984226 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:16.984279 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:16.987427 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:16.988568 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:16.988577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:16.988584 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:16.988790 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:16.989168 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:16.989172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:16.989174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:16.989799 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:16.990065 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:16.990620 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 17:00:16.991112 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":275,"threads_started":1}
I20260619 17:00:16.992997 1174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:17.079121 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 1050
2026-06-19T17:00:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65381 ms)
[----------] 30 tests from SecurityITest (309964 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:17 dist-test-slave-3ffd krb5kdc[1236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:17 dist-test-slave-3ffd krb5kdc[1236](info): set up 2 sockets
Jun 19 17:00:17 dist-test-slave-3ffd krb5kdc[1236](info): commencing operation
krb5kdc: starting...
W20260619 17:00:18.200595 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.008s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:18 dist-test-slave-3ffd krb5kdc[1236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:00:18.468789 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:39121
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:18.826262 1249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:18.826597 1249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:18.826771 1249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:18.835189 1249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:18.835340 1249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:18.835433 1249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:18.835549 1249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:18.835630 1249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:18.846431 1249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:18.847903 1249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:18.849386 1249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:18.862811 1254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:18.862845 1255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:18.862802 1257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:18.863637 1249 server_base.cc:1034] running on GCE node
I20260619 17:00:18.864692 1249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:18.866300 1249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:18.867555 1249 hybrid_clock.cc:648] HybridClock initialized: now 1781888418867483 us; error 61 us; skew 500 ppm
I20260619 17:00:18.867997 1249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:18 dist-test-slave-3ffd krb5kdc[1236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:18.871439 1249 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:00:18.873181 1249 webserver.cc:460] Webserver started at http://127.16.48.62:41151/ using document root <none> and password file <none>
I20260619 17:00:18.873919 1249 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:18.874028 1249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:18.874312 1249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:18.877599 1249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "b5aa8d33426c4628b1a55c8dacea6887"
format_stamp: "Formatted at 2026-06-19 17:00:18 on dist-test-slave-3ffd"
server_key: "228effb5003a5fa04d03bbe5582cd767"
server_key_iv: "40ffe938e49987cf544196a9cfcd9312"
server_key_version: "encryptionkey@0"
I20260619 17:00:18.878401 1249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "b5aa8d33426c4628b1a55c8dacea6887"
format_stamp: "Formatted at 2026-06-19 17:00:18 on dist-test-slave-3ffd"
server_key: "228effb5003a5fa04d03bbe5582cd767"
server_key_iv: "40ffe938e49987cf544196a9cfcd9312"
server_key_version: "encryptionkey@0"
I20260619 17:00:18.884245 1249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 17:00:18.888242 1264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:18.889809 1249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 17:00:18.890046 1249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "b5aa8d33426c4628b1a55c8dacea6887"
format_stamp: "Formatted at 2026-06-19 17:00:18 on dist-test-slave-3ffd"
server_key: "228effb5003a5fa04d03bbe5582cd767"
server_key_iv: "40ffe938e49987cf544196a9cfcd9312"
server_key_version: "encryptionkey@0"
I20260619 17:00:18.890291 1249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:18.930181 1249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:18.935241 1249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:18.935563 1249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:18.955847 1249 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:39121
I20260619 17:00:18.955868 1315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:39121 every 8 connection(s)
I20260619 17:00:18.957473 1249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:18.961791 1316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:18.965797 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 1249
I20260619 17:00:18.966041 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:18.966884 16576 external_mini_cluster.cc:1428] Setting key 08a4d59f2a10758a672991cf7206fd4d
I20260619 17:00:18.977092 1316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887: Bootstrap starting.
I20260619 17:00:18.981469 1316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:18.983007 1316 log.cc:824] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887: Log is configured to *not* fsync() on all Append() calls
Jun 19 17:00:18 dist-test-slave-3ffd krb5kdc[1236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:18.986557 1316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887: No bootstrap required, opened a new log
I20260619 17:00:18.997972 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:18.969228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.16.48.62:39121)
0619 17:00:18.970068 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:18.970088 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:18.970132 (+ 44us) server_negotiation.cc:407] Connection header received
0619 17:00:18.971887 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:18.971914 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:18.972459 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:18.972822 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:18.974910 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:18.976946 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:18.978917 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:18.979712 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:18.979883 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:18.985176 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:18.985185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:18.985202 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:18.985245 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:18.988636 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:18.989644 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:18.989659 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:18.989669 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:18.989939 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:18.990423 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:18.990434 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:18.990437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:18.991220 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:18.991535 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:18.995238 (+ 3703us) server_negotiation.cc:299] Negotiation successful
0619 17:00:18.996256 (+ 1018us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":399,"threads_started":1}
I20260619 17:00:19.003072 1316 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5aa8d33426c4628b1a55c8dacea6887" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39121 } }
I20260619 17:00:19.003485 1316 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:19.003603 1316 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5aa8d33426c4628b1a55c8dacea6887, State: Initialized, Role: FOLLOWER
I20260619 17:00:19.004310 1316 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5aa8d33426c4628b1a55c8dacea6887" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39121 } }
I20260619 17:00:19.004598 1316 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:19.004750 1316 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:19.004931 1316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:19.007452 1316 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5aa8d33426c4628b1a55c8dacea6887" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39121 } }
I20260619 17:00:19.008208 1316 leader_election.cc:302] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5aa8d33426c4628b1a55c8dacea6887; no voters:
I20260619 17:00:19.009276 1316 leader_election.cc:288] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:19.009430 1321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:19.011672 1316 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:19.012537 1321 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [term 1 LEADER]: Becoming Leader. State: Replica: b5aa8d33426c4628b1a55c8dacea6887, State: Running, Role: LEADER
I20260619 17:00:19.013432 1321 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5aa8d33426c4628b1a55c8dacea6887" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39121 } }
I20260619 17:00:19.022796 1322 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5aa8d33426c4628b1a55c8dacea6887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5aa8d33426c4628b1a55c8dacea6887" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39121 } } }
I20260619 17:00:19.023329 1322 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:19.024008 1323 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5aa8d33426c4628b1a55c8dacea6887. Latest consensus state: current_term: 1 leader_uuid: "b5aa8d33426c4628b1a55c8dacea6887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5aa8d33426c4628b1a55c8dacea6887" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39121 } } }
I20260619 17:00:19.024384 1323 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:19.026520 1330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:19.034614 1330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:19.057895 1330 catalog_manager.cc:1347] Generated new cluster ID: dae5a254c6764557bd2c98c46fff1ae1
I20260619 17:00:19.058014 1330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:19.084209 1330 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:19.086333 1330 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:19.093613 1340 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 17:00:19.106609 1330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887: Generated new TSK 0
I20260619 17:00:19.107815 1330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 17:00:19.207619 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:39121
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:19.629034 1343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:19.629348 1343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:19.629513 1343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:19.637279 1343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:19.637413 1343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:19.637614 1343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 17:00:19.647819 1343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:39121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:19.649262 1343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:19.650607 1343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:19.664554 1348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:19.664518 1349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:19.664734 1351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:19.665483 1343 server_base.cc:1034] running on GCE node
I20260619 17:00:19.666152 1343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:19.667161 1343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:19.668407 1343 hybrid_clock.cc:648] HybridClock initialized: now 1781888419668349 us; error 45 us; skew 500 ppm
I20260619 17:00:19.668869 1343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:19 dist-test-slave-3ffd krb5kdc[1236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888419, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:19.672324 1343 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 17:00:19.674110 1343 webserver.cc:460] Webserver started at http://127.16.48.1:38025/ using document root <none> and password file <none>
I20260619 17:00:19.674929 1343 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:19.675098 1343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:19.675439 1343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:19.678751 1343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "9759eeaa369747c18936efe7a4c7098b"
format_stamp: "Formatted at 2026-06-19 17:00:19 on dist-test-slave-3ffd"
server_key: "d8753d99a20a99cc3fee942c37b0d1db"
server_key_iv: "e2d359bd280c5b48d22fcafe9176c88c"
server_key_version: "encryptionkey@0"
I20260619 17:00:19.679663 1343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "9759eeaa369747c18936efe7a4c7098b"
format_stamp: "Formatted at 2026-06-19 17:00:19 on dist-test-slave-3ffd"
server_key: "d8753d99a20a99cc3fee942c37b0d1db"
server_key_iv: "e2d359bd280c5b48d22fcafe9176c88c"
server_key_version: "encryptionkey@0"
I20260619 17:00:19.685640 1343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 17:00:19.689858 1358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:19.691552 1343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 17:00:19.691815 1343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "9759eeaa369747c18936efe7a4c7098b"
format_stamp: "Formatted at 2026-06-19 17:00:19 on dist-test-slave-3ffd"
server_key: "d8753d99a20a99cc3fee942c37b0d1db"
server_key_iv: "e2d359bd280c5b48d22fcafe9176c88c"
server_key_version: "encryptionkey@0"
I20260619 17:00:19.692080 1343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:19.718017 1343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:19.722728 1343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:19.723086 1343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:19.724185 1343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:19.725720 1343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:19.725829 1343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:19.725984 1343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:19.726063 1343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:19.761749 1343 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:42085
I20260619 17:00:19.761802 1470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:42085 every 8 connection(s)
I20260619 17:00:19.763129 1343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 17:00:19.764531 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 1343
I20260619 17:00:19.764763 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 17:00:19.765609 16576 external_mini_cluster.cc:1428] Setting key f25f17b38820b3e615c4be061d9afbf1
Jun 19 17:00:19 dist-test-slave-3ffd krb5kdc[1236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888419, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:19.787364 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:19.766269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:45651 (local address 127.16.48.62:39121)
0619 17:00:19.766755 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:19.766760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:19.768032 (+ 1272us) server_negotiation.cc:407] Connection header received
0619 17:00:19.770086 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:19.770096 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:19.770200 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:19.770335 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:19.772086 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:19.773481 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:19.775310 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:19.775974 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:19.776090 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:19.781078 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:19.781084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:19.781087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:19.781128 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:19.783847 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:19.784794 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:19.784800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:19.784808 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:19.784950 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:19.785456 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:19.785460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:19.785462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:19.785749 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:19.785911 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:19.786709 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 17:00:19.786969 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260619 17:00:19.788542 1474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:19.766964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:39121 (local address 127.16.48.1:45651)
0619 17:00:19.767841 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:19.767903 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:19.769760 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:19.770579 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:19.770589 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:19.771202 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:19.771853 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:19.771875 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:19.773746 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:19.773751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:19.775110 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:19.775126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:19.776290 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:19.776297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:19.776670 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:19.777622 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:00:19.777658 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:19.780786 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:19.784054 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:19.784065 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:19.784089 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:19.784656 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:19.785086 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:19.785091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:19.785094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:19.785316 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:19.785893 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:19.785902 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:19.786301 (+ 399us) client_negotiation.cc:769] Sending connection context
0619 17:00:19.786583 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 17:00:19.787193 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":371,"threads_started":1}
I20260619 17:00:19.790247 1471 heartbeater.cc:347] Connected to a master server at 127.16.48.62:39121
I20260619 17:00:19.790602 1471 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 17:00:19.791400 1471 heartbeater.cc:511] Master 127.16.48.62:39121 requested a full tablet report, sending...
I20260619 17:00:19.793892 1281 ts_manager.cc:194] Registered new tserver with Master: 9759eeaa369747c18936efe7a4c7098b (127.16.48.1:42085)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 17:00:19.795526 1281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:45651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 17:00:19.825920 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:39121
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:20.183480 1477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:20.183813 1477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:20.183987 1477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:20.191756 1477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:20.191893 1477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:20.192098 1477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 17:00:20.202919 1477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:39121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:20.204371 1477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:20.205883 1477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:20.219748 1482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:20.219725 1485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:20.219780 1483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:20.220692 1477 server_base.cc:1034] running on GCE node
I20260619 17:00:20.221401 1477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:20.222397 1477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:20.223610 1477 hybrid_clock.cc:648] HybridClock initialized: now 1781888420223537 us; error 67 us; skew 500 ppm
I20260619 17:00:20.223946 1477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:20 dist-test-slave-3ffd krb5kdc[1236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888420, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:20.233153 1477 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 17:00:20.234802 1477 webserver.cc:460] Webserver started at http://127.16.48.2:42821/ using document root <none> and password file <none>
I20260619 17:00:20.235601 1477 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:20.235749 1477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:20.236083 1477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:20.239423 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "a74f9ebc17a1429781573dbd99a2bf84"
format_stamp: "Formatted at 2026-06-19 17:00:20 on dist-test-slave-3ffd"
server_key: "db998193cdca2366191bf6a4e0e75a90"
server_key_iv: "92aebc9f5e76d82e0a918b03c8f9cc99"
server_key_version: "encryptionkey@0"
I20260619 17:00:20.240315 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "a74f9ebc17a1429781573dbd99a2bf84"
format_stamp: "Formatted at 2026-06-19 17:00:20 on dist-test-slave-3ffd"
server_key: "db998193cdca2366191bf6a4e0e75a90"
server_key_iv: "92aebc9f5e76d82e0a918b03c8f9cc99"
server_key_version: "encryptionkey@0"
I20260619 17:00:20.246166 1477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 17:00:20.249982 1492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:20.251499 1477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 17:00:20.251710 1477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "a74f9ebc17a1429781573dbd99a2bf84"
format_stamp: "Formatted at 2026-06-19 17:00:20 on dist-test-slave-3ffd"
server_key: "db998193cdca2366191bf6a4e0e75a90"
server_key_iv: "92aebc9f5e76d82e0a918b03c8f9cc99"
server_key_version: "encryptionkey@0"
I20260619 17:00:20.251935 1477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:20.269738 1477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:20.274586 1477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:20.274909 1477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:20.275983 1477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:20.277477 1477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:20.277588 1477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:20.277748 1477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:20.277823 1477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:20.313627 1477 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:41061
I20260619 17:00:20.313678 1604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:41061 every 8 connection(s)
I20260619 17:00:20.315021 1477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 17:00:20.323119 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 1477
I20260619 17:00:20.323318 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 17:00:20.323822 16576 external_mini_cluster.cc:1428] Setting key f1b3abb9e7e0094c3331dc8ecacd70ba
Jun 19 17:00:20 dist-test-slave-3ffd krb5kdc[1236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888420, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:20.336045 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.317731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:56741 (local address 127.16.48.62:39121)
0619 17:00:20.318165 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:20.318170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:20.319308 (+ 1138us) server_negotiation.cc:407] Connection header received
0619 17:00:20.321267 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:20.321272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:20.321348 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:20.321452 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:20.323093 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.323924 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.325614 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.326034 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.326177 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:20.329949 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:20.329955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:20.329958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:20.329995 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:20.332754 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:20.333587 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:20.333591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:20.333593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:20.333719 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:20.334175 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:20.334179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:20.334181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:20.334466 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:20.334636 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:20.335387 (+ 751us) server_negotiation.cc:299] Negotiation successful
0619 17:00:20.335650 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
I20260619 17:00:20.337281 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.318329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:39121 (local address 127.16.48.2:56741)
0619 17:00:20.319130 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:20.319176 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:20.320977 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:20.321680 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:20.321689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:20.322253 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:20.322879 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.322898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.324104 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.324108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:20.325445 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.325459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.326284 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.326291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:20.326544 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:20.327292 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:00:20.327317 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:20.329664 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:20.332923 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:20.332933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:20.332954 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:20.333432 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:20.333834 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:20.333839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:20.333843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:20.334044 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:20.334596 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:20.334605 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:20.334976 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 17:00:20.335290 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 17:00:20.335865 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":330,"threads_started":1}
I20260619 17:00:20.338871 1605 heartbeater.cc:347] Connected to a master server at 127.16.48.62:39121
I20260619 17:00:20.339279 1605 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:20.340324 1605 heartbeater.cc:511] Master 127.16.48.62:39121 requested a full tablet report, sending...
I20260619 17:00:20.342177 1281 ts_manager.cc:194] Registered new tserver with Master: a74f9ebc17a1429781573dbd99a2bf84 (127.16.48.2:41061)
I20260619 17:00:20.343499 1281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:56741
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 17:00:20.383553 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:39121
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:20.740278 1611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:20.740583 1611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:20.740679 1611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:20.748361 1611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:20.748490 1611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:20.748608 1611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 17:00:20.759528 1611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:39121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:20.760967 1611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:20.762377 1611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:20.776971 1617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:20.776914 1619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:20.776995 1616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:20.778069 1611 server_base.cc:1034] running on GCE node
I20260619 17:00:20.778779 1611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:20.779860 1611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:20.781160 1611 hybrid_clock.cc:648] HybridClock initialized: now 1781888420781108 us; error 49 us; skew 500 ppm
I20260619 17:00:20.781541 1611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:20 dist-test-slave-3ffd krb5kdc[1236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888420, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:20.785120 1611 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 17:00:20.786922 1611 webserver.cc:460] Webserver started at http://127.16.48.3:36757/ using document root <none> and password file <none>
I20260619 17:00:20.787742 1611 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:20.787909 1611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:20.788345 1611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:20.791899 1611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293"
format_stamp: "Formatted at 2026-06-19 17:00:20 on dist-test-slave-3ffd"
server_key: "63e2a6caa553c2ecb023094b273fca65"
server_key_iv: "c27cb6b608d6aee77f0312be4e945016"
server_key_version: "encryptionkey@0"
I20260619 17:00:20.792939 1611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293"
format_stamp: "Formatted at 2026-06-19 17:00:20 on dist-test-slave-3ffd"
server_key: "63e2a6caa553c2ecb023094b273fca65"
server_key_iv: "c27cb6b608d6aee77f0312be4e945016"
server_key_version: "encryptionkey@0"
I20260619 17:00:20.798902 1471 heartbeater.cc:503] Master 127.16.48.62:39121 was elected leader, sending a full tablet report...
I20260619 17:00:20.799368 1611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 17:00:20.803622 1626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:20.805219 1611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 17:00:20.805440 1611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293"
format_stamp: "Formatted at 2026-06-19 17:00:20 on dist-test-slave-3ffd"
server_key: "63e2a6caa553c2ecb023094b273fca65"
server_key_iv: "c27cb6b608d6aee77f0312be4e945016"
server_key_version: "encryptionkey@0"
I20260619 17:00:20.805688 1611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:20.826128 1611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:20.831141 1611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:20.831483 1611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:20.832679 1611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:20.834208 1611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:20.834324 1611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:20.834478 1611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:20.834555 1611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:20.870132 1611 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:40351
I20260619 17:00:20.870178 1738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:40351 every 8 connection(s)
I20260619 17:00:20.871546 1611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 17:00:20.880164 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 1611
I20260619 17:00:20.880365 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 17:00:20.880820 16576 external_mini_cluster.cc:1428] Setting key 49c88ce08f79e8c69a0923610d15e04f
Jun 19 17:00:20 dist-test-slave-3ffd krb5kdc[1236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888420, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:20.892970 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.874298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:32861 (local address 127.16.48.62:39121)
0619 17:00:20.874800 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:20.874805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:20.875722 (+ 917us) server_negotiation.cc:407] Connection header received
0619 17:00:20.877744 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:20.877749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:20.877846 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:20.877969 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:20.879615 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.880602 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.882545 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.883068 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.883210 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:20.886964 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:20.886970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:20.886973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:20.887013 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:20.889581 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:20.890424 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:20.890435 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:20.890437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:20.890574 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:20.891042 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:20.891047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:20.891055 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:20.891337 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:20.891507 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:20.892306 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 17:00:20.892566 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":286,"threads_started":1}
I20260619 17:00:20.894104 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.874760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:39121 (local address 127.16.48.3:32861)
0619 17:00:20.875507 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:20.875548 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:20.877423 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:20.878240 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:20.878250 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:20.878806 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:20.879421 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.879443 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.880768 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.880772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:20.882394 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.882409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.883325 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.883333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:20.883588 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:20.884350 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:00:20.884378 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:20.886692 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:20.889755 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:20.889765 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:20.889786 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:20.890266 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:20.890696 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:20.890702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:20.890706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:20.890913 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:20.891475 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:20.891483 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:20.891871 (+ 388us) client_negotiation.cc:769] Sending connection context
0619 17:00:20.892207 (+ 336us) client_negotiation.cc:240] Negotiation successful
0619 17:00:20.892802 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
I20260619 17:00:20.895896 1739 heartbeater.cc:347] Connected to a master server at 127.16.48.62:39121
I20260619 17:00:20.896289 1739 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:20.897152 1739 heartbeater.cc:511] Master 127.16.48.62:39121 requested a full tablet report, sending...
I20260619 17:00:20.898639 1280 ts_manager.cc:194] Registered new tserver with Master: c33cd6d6eb9f4a2cbefb9d5e5c229293 (127.16.48.3:40351)
I20260619 17:00:20.899511 1280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:32861
I20260619 17:00:20.909772 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 17:00:20.928715 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.914945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.16.48.62:39121)
0619 17:00:20.915159 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:20.915165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:20.915637 (+ 472us) server_negotiation.cc:407] Connection header received
0619 17:00:20.915793 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:20.915797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:20.915867 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:20.915964 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:20.917527 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.918291 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.919689 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.920257 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.920413 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:20.923104 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:20.923109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:20.923112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:20.923152 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:20.925834 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:20.926633 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:20.926637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:20.926640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:20.926810 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:20.927198 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:20.927201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:20.927203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:20.927582 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:20.927786 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:20.928251 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 17:00:20.928462 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 17:00:20.933413 1280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 17:00:20.940949 1280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 17:00:20.990613 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.954742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:40351 (local address 127.0.0.1:53182)
0619 17:00:20.970488 (+ 15746us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:20.970537 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:20.970808 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:20.972774 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:20.972787 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:20.972838 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:20.973409 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.973428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.978887 (+ 5459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.978897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:20.982564 (+ 3667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.982592 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.988995 (+ 6403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.989005 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:20.989368 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:20.989464 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 17:00:20.989649 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 17:00:20.989976 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15432,"thread_start_us":15182,"threads_started":1}
I20260619 17:00:20.992022 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.954987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.16.48.3:40351)
0619 17:00:20.970417 (+ 15430us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:20.970430 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:20.970725 (+ 295us) server_negotiation.cc:407] Connection header received
0619 17:00:20.971195 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:20.971208 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:20.971981 (+ 773us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:20.972313 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:20.973699 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.977764 (+ 4065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.983998 (+ 6234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.986443 (+ 2445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.986625 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:20.991315 (+ 4690us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:20.991414 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 17:00:20.991632 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15186,"thread_start_us":15446,"threads_started":1}
I20260619 17:00:20.992074 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.955260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41061 (local address 127.0.0.1:46302)
0619 17:00:20.975477 (+ 20217us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:20.975498 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:20.975642 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:20.983751 (+ 8109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:20.983760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:20.983799 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:20.984309 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.984329 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.986387 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.986394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:20.988785 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.988804 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.991183 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.991193 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:20.991407 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:20.991443 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 17:00:20.991534 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 17:00:20.991699 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20086,"thread_start_us":19942,"threads_started":1}
I20260619 17:00:20.992436 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.955351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.16.48.2:41061)
0619 17:00:20.974872 (+ 19521us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:20.974881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:20.980312 (+ 5431us) server_negotiation.cc:407] Connection header received
0619 17:00:20.982876 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:20.982886 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:20.983223 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:20.983467 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:20.984540 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.986200 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.989014 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.990626 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.990764 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:20.990978 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:20.991679 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 17:00:20.991871 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19333,"thread_start_us":19526,"threads_started":1}
I20260619 17:00:20.996759 1540 tablet_service.cc:1467] Processing CreateTablet for tablet e77f0a910c254d21af37040578342a5b (DEFAULT_TABLE table=test-table [id=5bd45e0ea53744459059fd4da382dc9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:20.996763 1674 tablet_service.cc:1467] Processing CreateTablet for tablet e77f0a910c254d21af37040578342a5b (DEFAULT_TABLE table=test-table [id=5bd45e0ea53744459059fd4da382dc9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:20.998493 1674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77f0a910c254d21af37040578342a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:20.998790 1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77f0a910c254d21af37040578342a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:21.001528 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.975869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42085 (local address 127.0.0.1:60474)
0619 17:00:20.991575 (+ 15706us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:20.991596 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:20.991843 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:20.992990 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:20.992997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:20.993029 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:20.993385 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.993398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.995597 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.995603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:20.997791 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:20.997815 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.000557 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.000565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:21.000800 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.000851 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 17:00:21.000964 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 17:00:21.001129 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15522,"thread_start_us":15339,"threads_started":1}
I20260619 17:00:21.002537 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:20.975889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.16.48.1:42085)
0619 17:00:20.991011 (+ 15122us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:20.991023 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:20.991709 (+ 686us) server_negotiation.cc:407] Connection header received
0619 17:00:20.992048 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:20.992061 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:20.992711 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:20.992911 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:20.993610 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.995392 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:20.998054 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:20.999841 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.000038 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.000342 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:21.001827 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0619 17:00:21.002058 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5543,"thread_start_us":6466,"threads_started":1}
I20260619 17:00:21.007174 1406 tablet_service.cc:1467] Processing CreateTablet for tablet e77f0a910c254d21af37040578342a5b (DEFAULT_TABLE table=test-table [id=5bd45e0ea53744459059fd4da382dc9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:21.009541 1406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77f0a910c254d21af37040578342a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:21.012569 1758 tablet_bootstrap.cc:492] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: Bootstrap starting.
I20260619 17:00:21.014803 1759 tablet_bootstrap.cc:492] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: Bootstrap starting.
I20260619 17:00:21.017021 1758 tablet_bootstrap.cc:654] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:21.018560 1758 log.cc:824] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:21.018654 1759 tablet_bootstrap.cc:654] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:21.019888 1759 log.cc:824] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:21.022127 1758 tablet_bootstrap.cc:492] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: No bootstrap required, opened a new log
I20260619 17:00:21.022457 1758 ts_tablet_manager.cc:1397] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260619 17:00:21.023162 1759 tablet_bootstrap.cc:492] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: No bootstrap required, opened a new log
I20260619 17:00:21.023434 1759 ts_tablet_manager.cc:1397] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 17:00:21.024755 1762 tablet_bootstrap.cc:492] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: Bootstrap starting.
I20260619 17:00:21.029178 1762 tablet_bootstrap.cc:654] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:21.030371 1762 log.cc:824] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:21.033504 1762 tablet_bootstrap.cc:492] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: No bootstrap required, opened a new log
I20260619 17:00:21.033836 1762 ts_tablet_manager.cc:1397] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 17:00:21.034472 1759 raft_consensus.cc:348] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.034463 1758 raft_consensus.cc:348] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.034967 1758 raft_consensus.cc:374] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:21.034984 1759 raft_consensus.cc:374] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:21.035081 1758 raft_consensus.cc:729] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74f9ebc17a1429781573dbd99a2bf84, State: Initialized, Role: FOLLOWER
I20260619 17:00:21.035086 1759 raft_consensus.cc:729] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33cd6d6eb9f4a2cbefb9d5e5c229293, State: Initialized, Role: FOLLOWER
I20260619 17:00:21.035826 1758 consensus_queue.cc:260] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.035826 1759 consensus_queue.cc:260] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.037177 1739 heartbeater.cc:503] Master 127.16.48.62:39121 was elected leader, sending a full tablet report...
I20260619 17:00:21.037719 1605 heartbeater.cc:503] Master 127.16.48.62:39121 was elected leader, sending a full tablet report...
I20260619 17:00:21.038545 1758 ts_tablet_manager.cc:1428] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: Time spent starting tablet: real 0.016s user 0.017s sys 0.001s
I20260619 17:00:21.040143 1759 ts_tablet_manager.cc:1428] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20260619 17:00:21.045388 1762 raft_consensus.cc:348] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.045838 1762 raft_consensus.cc:374] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:21.045961 1762 raft_consensus.cc:729] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9759eeaa369747c18936efe7a4c7098b, State: Initialized, Role: FOLLOWER
I20260619 17:00:21.046573 1762 consensus_queue.cc:260] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.048324 1762 ts_tablet_manager.cc:1428] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
W20260619 17:00:21.067234 1606 tablet.cc:2369] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:21.106704 1766 raft_consensus.cc:482] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 17:00:21.107017 1766 raft_consensus.cc:504] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.108891 1766 leader_election.cc:288] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c33cd6d6eb9f4a2cbefb9d5e5c229293 (127.16.48.3:40351), a74f9ebc17a1429781573dbd99a2bf84 (127.16.48.2:41061)
I20260619 17:00:21.115137 1764 raft_consensus.cc:482] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 17:00:21.115511 1764 raft_consensus.cc:504] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.117457 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.109408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:50825 (local address 127.16.48.2:41061)
0619 17:00:21.109667 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:21.109676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:21.110432 (+ 756us) server_negotiation.cc:407] Connection header received
0619 17:00:21.110699 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:21.110707 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:21.110797 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:21.110872 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:21.111710 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.112645 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.114854 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.116205 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.116361 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.116476 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:21.116954 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 17:00:21.117101 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 17:00:21.117520 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.109341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:43377 (local address 127.16.48.3:40351)
0619 17:00:21.109681 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:21.109690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:21.110498 (+ 808us) server_negotiation.cc:407] Connection header received
0619 17:00:21.110780 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:21.110784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:21.110886 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:21.110999 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:21.111993 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.113075 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.115139 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.116369 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.116526 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.116658 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:21.117095 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 17:00:21.117247 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260619 17:00:21.117875 1764 leader_election.cc:288] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74f9ebc17a1429781573dbd99a2bf84 (127.16.48.2:41061), 9759eeaa369747c18936efe7a4c7098b (127.16.48.1:42085)
I20260619 17:00:21.117929 1767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.109262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41061 (local address 127.16.48.1:50825)
0619 17:00:21.110282 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:21.110302 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:21.110473 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:21.110916 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:21.110922 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:21.111028 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:21.111478 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:21.111492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.112812 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.112819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:21.114681 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:21.114694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.116473 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.116483 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:21.116751 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.116781 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 17:00:21.116921 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 17:00:21.117086 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":112,"thread_start_us":684,"threads_started":1}
I20260619 17:00:21.118269 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.109290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:40351 (local address 127.16.48.1:43377)
0619 17:00:21.110341 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:21.110378 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:21.110634 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:21.111044 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:21.111049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:21.111072 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:21.111626 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:21.111642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.113225 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.113229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:21.114941 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:21.114959 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.116673 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.116679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:21.116899 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.116931 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 17:00:21.117699 (+ 768us) client_negotiation.cc:240] Negotiation successful
0619 17:00:21.117848 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":54,"thread_start_us":708,"threads_started":1}
I20260619 17:00:21.119798 1560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77f0a910c254d21af37040578342a5b" candidate_uuid: "9759eeaa369747c18936efe7a4c7098b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74f9ebc17a1429781573dbd99a2bf84" is_pre_election: true
I20260619 17:00:21.119931 1694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77f0a910c254d21af37040578342a5b" candidate_uuid: "9759eeaa369747c18936efe7a4c7098b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" is_pre_election: true
I20260619 17:00:21.120481 1560 raft_consensus.cc:2454] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9759eeaa369747c18936efe7a4c7098b in term 0.
I20260619 17:00:21.120482 1694 raft_consensus.cc:2454] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9759eeaa369747c18936efe7a4c7098b in term 0.
I20260619 17:00:21.121320 1361 leader_election.cc:302] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9759eeaa369747c18936efe7a4c7098b, c33cd6d6eb9f4a2cbefb9d5e5c229293; no voters:
I20260619 17:00:21.121856 1766 raft_consensus.cc:2781] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 17:00:21.122041 1766 raft_consensus.cc:482] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 17:00:21.122159 1766 raft_consensus.cc:3037] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 0 FOLLOWER]: Advancing to term 1
W20260619 17:00:21.123329 1740 tablet.cc:2369] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:21.125281 1766 raft_consensus.cc:504] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.126405 1766 leader_election.cc:288] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [CANDIDATE]: Term 1 election: Requested vote from peers c33cd6d6eb9f4a2cbefb9d5e5c229293 (127.16.48.3:40351), a74f9ebc17a1429781573dbd99a2bf84 (127.16.48.2:41061)
I20260619 17:00:21.126895 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.118117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:41061 (local address 127.16.48.3:41861)
0619 17:00:21.118328 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:21.118351 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:21.118473 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:21.119606 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:21.119613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:21.119650 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:21.119894 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:21.119911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.121786 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.121792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:21.124023 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:21.124039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.125891 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.125897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:21.126296 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.126335 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 17:00:21.126438 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 17:00:21.126662 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":97}
I20260619 17:00:21.126835 1694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77f0a910c254d21af37040578342a5b" candidate_uuid: "9759eeaa369747c18936efe7a4c7098b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293"
I20260619 17:00:21.127140 1694 raft_consensus.cc:3037] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:21.127492 1560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77f0a910c254d21af37040578342a5b" candidate_uuid: "9759eeaa369747c18936efe7a4c7098b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74f9ebc17a1429781573dbd99a2bf84"
I20260619 17:00:21.127799 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.118902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:41861 (local address 127.16.48.2:41061)
0619 17:00:21.119130 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:21.119137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:21.119159 (+ 22us) server_negotiation.cc:407] Connection header received
0619 17:00:21.119311 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:21.119317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:21.119420 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:21.120074 (+ 654us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:21.120303 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.121571 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.124286 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.125674 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.125854 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.125990 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:21.127305 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0619 17:00:21.127467 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 17:00:21.127755 1560 raft_consensus.cc:3037] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:21.128401 1559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77f0a910c254d21af37040578342a5b" candidate_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74f9ebc17a1429781573dbd99a2bf84" is_pre_election: true
I20260619 17:00:21.130558 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.121045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:54675 (local address 127.16.48.1:42085)
0619 17:00:21.121239 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:21.121245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:21.121261 (+ 16us) server_negotiation.cc:407] Connection header received
0619 17:00:21.121322 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:21.121327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:21.121395 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:21.121487 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:21.122473 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.123688 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.127597 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.129050 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.129206 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.129333 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:21.130090 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 17:00:21.130259 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 17:00:21.130877 1560 raft_consensus.cc:2454] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9759eeaa369747c18936efe7a4c7098b in term 1.
I20260619 17:00:21.130939 1694 raft_consensus.cc:2454] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9759eeaa369747c18936efe7a4c7098b in term 1.
I20260619 17:00:21.131430 1362 leader_election.cc:302] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9759eeaa369747c18936efe7a4c7098b, a74f9ebc17a1429781573dbd99a2bf84; no voters:
I20260619 17:00:21.131541 1769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.118506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:42085 (local address 127.16.48.3:54675)
0619 17:00:21.119336 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:21.119358 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:21.119480 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:21.121657 (+ 2177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:21.121662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:21.121702 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:21.122252 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:21.122268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.123939 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.123948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:21.127319 (+ 3371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:21.127338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.129582 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.129590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:21.129831 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.129875 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 17:00:21.130850 (+ 975us) client_negotiation.cc:240] Negotiation successful
0619 17:00:21.131045 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":35,"thread_start_us":471,"threads_started":1}
I20260619 17:00:21.131827 1766 raft_consensus.cc:2781] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:21.132491 1766 raft_consensus.cc:686] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 1 LEADER]: Becoming Leader. State: Replica: 9759eeaa369747c18936efe7a4c7098b, State: Running, Role: LEADER
I20260619 17:00:21.132828 1426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77f0a910c254d21af37040578342a5b" candidate_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9759eeaa369747c18936efe7a4c7098b" is_pre_election: true
I20260619 17:00:21.133109 1766 consensus_queue.cc:237] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } }
I20260619 17:00:21.134198 1630 leader_election.cc:302] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c33cd6d6eb9f4a2cbefb9d5e5c229293; no voters: 9759eeaa369747c18936efe7a4c7098b, a74f9ebc17a1429781573dbd99a2bf84
I20260619 17:00:21.134729 1764 raft_consensus.cc:2726] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 17:00:21.138417 1279 catalog_manager.cc:5495] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9759eeaa369747c18936efe7a4c7098b (127.16.48.1). New cstate: current_term: 1 leader_uuid: "9759eeaa369747c18936efe7a4c7098b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9759eeaa369747c18936efe7a4c7098b" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 42085 } health_report { overall_health: HEALTHY } } }
I20260619 17:00:21.185879 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.179860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.16.48.1:42085)
0619 17:00:21.180047 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:21.180052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:21.180072 (+ 20us) server_negotiation.cc:407] Connection header received
0619 17:00:21.180336 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:21.180342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:21.180484 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:21.180631 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:00:21.181016 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.181818 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.183290 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.183827 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.183960 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.184665 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:00:21.185241 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:00:21.185361 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:21.185454 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 17:00:21.185534 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 17:00:21.195003 1560 raft_consensus.cc:1261] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [term 1 FOLLOWER]: Refusing update from remote peer 9759eeaa369747c18936efe7a4c7098b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:21.195063 1694 raft_consensus.cc:1261] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 1 FOLLOWER]: Refusing update from remote peer 9759eeaa369747c18936efe7a4c7098b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:21.196023 1770 consensus_queue.cc:1035] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33cd6d6eb9f4a2cbefb9d5e5c229293" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:21.196503 1766 consensus_queue.cc:1035] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74f9ebc17a1429781573dbd99a2bf84" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:21.225885 1280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:21.226224 1280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:21.230495 1280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5aa8d33426c4628b1a55c8dacea6887: Sending DeleteTablet for 3 replicas of tablet e77f0a910c254d21af37040578342a5b
I20260619 17:00:21.231904 1540 tablet_service.cc:1514] Processing DeleteTablet for tablet e77f0a910c254d21af37040578342a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:21 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:46302
I20260619 17:00:21.231904 1674 tablet_service.cc:1514] Processing DeleteTablet for tablet e77f0a910c254d21af37040578342a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:21 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:53182
I20260619 17:00:21.232769 1406 tablet_service.cc:1514] Processing DeleteTablet for tablet e77f0a910c254d21af37040578342a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:21 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:60474
I20260619 17:00:21.232975 1783 tablet_replica.cc:330] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: stopping tablet replica
I20260619 17:00:21.233466 1783 raft_consensus.cc:2229] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:21.234037 1783 raft_consensus.cc:2258] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:21.234851 1784 tablet_replica.cc:330] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: stopping tablet replica
I20260619 17:00:21.234917 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 1343
I20260619 17:00:21.235277 1784 raft_consensus.cc:2229] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:21.236254 1784 raft_consensus.cc:2258] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:21.238600 1783 ts_tablet_manager.cc:1905] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:21.245980 1785 tablet_replica.cc:330] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: stopping tablet replica
I20260619 17:00:21.246407 1784 ts_tablet_manager.cc:1905] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:21.246769 1785 raft_consensus.cc:2229] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 1 LEADER]: Raft consensus shutting down.
I20260619 17:00:21.246879 1783 ts_tablet_manager.cc:1918] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:21.247066 1783 log.cc:1196] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/e77f0a910c254d21af37040578342a5b
I20260619 17:00:21.247674 1783 ts_tablet_manager.cc:1939] T e77f0a910c254d21af37040578342a5b P a74f9ebc17a1429781573dbd99a2bf84: Deleting consensus metadata
I20260619 17:00:21.247851 1785 raft_consensus.cc:2258] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:21.249780 1268 catalog_manager.cc:4841] TS a74f9ebc17a1429781573dbd99a2bf84 (127.16.48.2:41061): tablet e77f0a910c254d21af37040578342a5b (table test-table [id=5bd45e0ea53744459059fd4da382dc9e]) successfully deleted
I20260619 17:00:21.251206 1785 ts_tablet_manager.cc:1905] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:21.253825 1784 ts_tablet_manager.cc:1918] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:21.254026 1784 log.cc:1196] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/e77f0a910c254d21af37040578342a5b
I20260619 17:00:21.254628 1784 ts_tablet_manager.cc:1939] T e77f0a910c254d21af37040578342a5b P c33cd6d6eb9f4a2cbefb9d5e5c229293: Deleting consensus metadata
I20260619 17:00:21.256768 1267 catalog_manager.cc:4841] TS c33cd6d6eb9f4a2cbefb9d5e5c229293 (127.16.48.3:40351): tablet e77f0a910c254d21af37040578342a5b (table test-table [id=5bd45e0ea53744459059fd4da382dc9e]) successfully deleted
I20260619 17:00:21.259169 1785 ts_tablet_manager.cc:1918] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:21.259387 1785 log.cc:1196] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/e77f0a910c254d21af37040578342a5b
I20260619 17:00:21.260012 1785 ts_tablet_manager.cc:1939] T e77f0a910c254d21af37040578342a5b P 9759eeaa369747c18936efe7a4c7098b: Deleting consensus metadata
I20260619 17:00:21.262633 1268 catalog_manager.cc:4841] TS 9759eeaa369747c18936efe7a4c7098b (127.16.48.1:42085): tablet e77f0a910c254d21af37040578342a5b (table test-table [id=5bd45e0ea53744459059fd4da382dc9e]) successfully deleted
Jun 19 17:00:21 dist-test-slave-3ffd krb5kdc[1236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 17:00:21.271931 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.237282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.16.48.1:42085)
0619 17:00:21.237520 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:21.237534 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:21.252845 (+ 15311us) server_negotiation.cc:407] Connection header received
0619 17:00:21.253385 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:21.253392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:21.253507 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:21.253600 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:21.255603 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.256875 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.259165 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.259949 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.260105 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.265470 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:21.265485 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:21.265501 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:21.265556 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:21.268887 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:21.269723 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:21.269731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:21.269734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:21.269867 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:21.270259 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:21.270264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:21.270267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:21.270687 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:21.271026 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:21.271349 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 17:00:21.271596 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 17:00:21.272634 1466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:21.376089 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 1343
I20260619 17:00:21.394877 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 1477
Jun 19 17:00:21 dist-test-slave-3ffd krb5kdc[1236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 17:00:21.410969 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.395705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.16.48.2:41061)
0619 17:00:21.395893 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:21.395898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:21.395917 (+ 19us) server_negotiation.cc:407] Connection header received
0619 17:00:21.396116 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:21.396172 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:21.396251 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:21.396349 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:21.397648 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.398348 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.399732 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.400227 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.400335 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.404856 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:21.404874 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:21.404891 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:21.404935 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:21.407977 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:21.408806 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:21.408815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:21.408817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:21.408974 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:21.409324 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:21.409328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:21.409330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:21.409830 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:21.410111 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:21.410338 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 17:00:21.410582 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 17:00:21.411856 1600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:21.504776 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 1477
I20260619 17:00:21.522818 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 1611
Jun 19 17:00:21 dist-test-slave-3ffd krb5kdc[1236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 17:00:21.539336 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:21.523641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.16.48.3:40351)
0619 17:00:21.523826 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:21.523833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:21.523856 (+ 23us) server_negotiation.cc:407] Connection header received
0619 17:00:21.523976 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:21.523980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:21.524050 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:21.524184 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:21.525536 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.526309 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.527756 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:21.528278 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:21.528427 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:21.533181 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:21.533199 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:21.533216 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:21.533262 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:21.536422 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:21.537180 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:21.537188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:21.537190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:21.537320 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:21.537717 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:21.537722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:21.537730 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:21.538110 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:21.538403 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:21.538737 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 17:00:21.538963 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 17:00:21.540087 1734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:21.631975 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 1611
I20260619 17:00:21.650244 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 1249
I20260619 17:00:21.651436 1311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:21.764542 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 1249
2026-06-19T17:00:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4693 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T17:00:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:21Z Disabled control of system clock
I20260619 17:00:21.850800 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36509
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:36509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 17:00:22.211622 1794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:22.211969 1794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:22.212113 1794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:22.219944 1794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:22.220086 1794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:22.220187 1794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:22.220292 1794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:22.220368 1794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:22.230371 1794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:36509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:22.231815 1794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:22.233762 1794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:22.246418 1799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:22.246332 1802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:22.246387 1800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:22.247290 1794 server_base.cc:1034] running on GCE node
I20260619 17:00:22.248194 1794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:22.249773 1794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:22.251017 1794 hybrid_clock.cc:648] HybridClock initialized: now 1781888422250978 us; error 57 us; skew 500 ppm
I20260619 17:00:22.251394 1794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:22.253929 1794 webserver.cc:460] Webserver started at http://127.16.48.62:45939/ using document root <none> and password file <none>
I20260619 17:00:22.254695 1794 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:22.254830 1794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:22.255162 1794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:22.258373 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e"
format_stamp: "Formatted at 2026-06-19 17:00:22 on dist-test-slave-3ffd"
server_key: "6fc893f3ef89505a222f409927708fc4"
server_key_iv: "9a4cfbf4603ee1dc07c502c5c6c99ac3"
server_key_version: "encryptionkey@0"
I20260619 17:00:22.259296 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e"
format_stamp: "Formatted at 2026-06-19 17:00:22 on dist-test-slave-3ffd"
server_key: "6fc893f3ef89505a222f409927708fc4"
server_key_iv: "9a4cfbf4603ee1dc07c502c5c6c99ac3"
server_key_version: "encryptionkey@0"
I20260619 17:00:22.265292 1794 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 17:00:22.269333 1808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:22.270920 1794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 17:00:22.271171 1794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e"
format_stamp: "Formatted at 2026-06-19 17:00:22 on dist-test-slave-3ffd"
server_key: "6fc893f3ef89505a222f409927708fc4"
server_key_iv: "9a4cfbf4603ee1dc07c502c5c6c99ac3"
server_key_version: "encryptionkey@0"
I20260619 17:00:22.271435 1794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:22.294729 1794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:22.295708 1794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:22.296072 1794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:22.315264 1794 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:36509
I20260619 17:00:22.315289 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:36509 every 8 connection(s)
I20260619 17:00:22.316854 1794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:22.317252 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 1794
I20260619 17:00:22.317417 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:22.317936 16576 external_mini_cluster.cc:1428] Setting key 45e2b9d9c5a37a7008056ab30d5aa5ee
I20260619 17:00:22.321858 1860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:22.332075 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:22.319971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.16.48.62:36509)
0619 17:00:22.321064 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:22.321077 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:22.321125 (+ 48us) server_negotiation.cc:407] Connection header received
0619 17:00:22.322903 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:22.322929 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:22.323419 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:22.323829 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:22.324495 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:22.325875 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:22.327378 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:22.328104 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:22.328365 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:22.329576 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:22.329584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:22.329600 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:22.330205 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:22.330293 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:22.330315 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:22.330476 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:22.330715 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 17:00:22.331051 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":380,"threads_started":1}
I20260619 17:00:22.333853 1860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e: Bootstrap starting.
I20260619 17:00:22.337420 1860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:22.338536 1860 log.cc:824] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:22.342126 1860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e: No bootstrap required, opened a new log
I20260619 17:00:22.350184 1860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36509 } }
I20260619 17:00:22.350648 1860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:22.350764 1860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4b66f2e4084f1cb7431ffc6165ec1e, State: Initialized, Role: FOLLOWER
I20260619 17:00:22.351405 1860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36509 } }
I20260619 17:00:22.351677 1860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:22.351835 1860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:22.352030 1860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:22.354398 1860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36509 } }
I20260619 17:00:22.355079 1860 leader_election.cc:302] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e4b66f2e4084f1cb7431ffc6165ec1e; no voters:
I20260619 17:00:22.355770 1860 leader_election.cc:288] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:22.355942 1865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:22.356784 1865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [term 1 LEADER]: Becoming Leader. State: Replica: 1e4b66f2e4084f1cb7431ffc6165ec1e, State: Running, Role: LEADER
I20260619 17:00:22.357406 1865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36509 } }
I20260619 17:00:22.358276 1860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:22.360414 1866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36509 } } }
I20260619 17:00:22.361505 1866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:22.362763 1867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e4b66f2e4084f1cb7431ffc6165ec1e. Latest consensus state: current_term: 1 leader_uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b66f2e4084f1cb7431ffc6165ec1e" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36509 } } }
I20260619 17:00:22.363145 1867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:22.363538 1872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:22.369702 1872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:22.380918 1872 catalog_manager.cc:1347] Generated new cluster ID: aa5c1bd8dc1149cb8e8b7447e06e2b21
I20260619 17:00:22.381057 1872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:22.399672 1872 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:22.401161 1872 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:22.417951 1872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e: Generated new TSK 0
I20260619 17:00:22.418927 1872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:22.423892 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36509
--builtin_ntp_servers=127.16.48.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 17:00:22.806354 1884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:22.806687 1884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:22.806777 1884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:22.815062 1884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:22.815209 1884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:22.815335 1884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 17:00:22.826633 1884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:22.827996 1884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:22.829870 1884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:22.844563 1889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:22.844655 1890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:22.844599 1892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:22.850476 1884 server_base.cc:1034] running on GCE node
I20260619 17:00:22.851195 1884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:22.852370 1884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:22.853706 1884 hybrid_clock.cc:648] HybridClock initialized: now 1781888422853663 us; error 46 us; skew 500 ppm
I20260619 17:00:22.854106 1884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:22.856890 1884 webserver.cc:460] Webserver started at http://127.16.48.1:44265/ using document root <none> and password file <none>
I20260619 17:00:22.857698 1884 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:22.857841 1884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:22.858180 1884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:22.861971 1884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "2d9499f3a21b4603aec33ffae8ddd21a"
format_stamp: "Formatted at 2026-06-19 17:00:22 on dist-test-slave-3ffd"
server_key: "abf899cbf898449a7ed5d78bf80fe010"
server_key_iv: "e0e202b2692096831a57f9a6fd46f22e"
server_key_version: "encryptionkey@0"
I20260619 17:00:22.863092 1884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "2d9499f3a21b4603aec33ffae8ddd21a"
format_stamp: "Formatted at 2026-06-19 17:00:22 on dist-test-slave-3ffd"
server_key: "abf899cbf898449a7ed5d78bf80fe010"
server_key_iv: "e0e202b2692096831a57f9a6fd46f22e"
server_key_version: "encryptionkey@0"
I20260619 17:00:22.869355 1884 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 17:00:22.873276 1898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:22.874951 1884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 17:00:22.875174 1884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "2d9499f3a21b4603aec33ffae8ddd21a"
format_stamp: "Formatted at 2026-06-19 17:00:22 on dist-test-slave-3ffd"
server_key: "abf899cbf898449a7ed5d78bf80fe010"
server_key_iv: "e0e202b2692096831a57f9a6fd46f22e"
server_key_version: "encryptionkey@0"
I20260619 17:00:22.875447 1884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:22.911240 1884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:22.912154 1884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:22.912415 1884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:22.913621 1884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:22.915218 1884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:22.915313 1884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:22.915390 1884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:22.915446 1884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:22.950947 1884 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:45779
I20260619 17:00:22.951005 2010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:45779 every 8 connection(s)
I20260619 17:00:22.952456 1884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 17:00:22.955581 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 1884
I20260619 17:00:22.955766 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 17:00:22.956341 16576 external_mini_cluster.cc:1428] Setting key 81d2b3e1d2b26eb054fffda1d225ca3a
I20260619 17:00:22.959875 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36509
--builtin_ntp_servers=127.16.48.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 17:00:22.966995 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:22.955215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:56201 (local address 127.16.48.62:36509)
0619 17:00:22.955823 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:22.955832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:22.957480 (+ 1648us) server_negotiation.cc:407] Connection header received
0619 17:00:22.959897 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:22.959903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:22.960038 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:22.960180 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:22.961363 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:22.962096 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:22.963907 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:22.964389 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:22.964572 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:22.966089 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:22.966094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:22.966098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:22.966119 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:22.966167 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:22.966180 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:22.966274 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:22.966520 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 17:00:22.966634 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":294,"threads_started":1}
I20260619 17:00:22.968111 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:22.956031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36509 (local address 127.16.48.1:56201)
0619 17:00:22.957310 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:22.957364 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:22.959599 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:22.960463 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:22.960472 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:22.960583 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:22.961185 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:22.961203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:22.962277 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:22.962281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:22.963741 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:22.963755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:22.964682 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:22.964692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:22.964952 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:22.965720 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 17:00:22.965722 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:22.965768 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 17:00:22.965863 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:22.966290 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:22.966309 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:22.966316 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 17:00:22.966506 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 17:00:22.966965 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":436,"threads_started":1}
I20260619 17:00:22.969892 2011 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36509
I20260619 17:00:22.970209 2011 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:22.971047 2011 heartbeater.cc:511] Master 127.16.48.62:36509 requested a full tablet report, sending...
I20260619 17:00:22.973294 1825 ts_manager.cc:194] Registered new tserver with Master: 2d9499f3a21b4603aec33ffae8ddd21a (127.16.48.1:45779)
I20260619 17:00:22.974880 1825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.1:56201
W20260619 17:00:23.319422 2015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:23.319798 2015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:23.319962 2015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:23.327816 2015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:23.328006 2015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:23.328253 2015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 17:00:23.339141 2015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:23.340638 2015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:23.342522 2015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:23.357347 2020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:23.357393 2021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:23.357363 2023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:23.358388 2015 server_base.cc:1034] running on GCE node
I20260619 17:00:23.359098 2015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:23.360286 2015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:23.361529 2015 hybrid_clock.cc:648] HybridClock initialized: now 1781888423361448 us; error 66 us; skew 500 ppm
I20260619 17:00:23.361929 2015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:23.364928 2015 webserver.cc:460] Webserver started at http://127.16.48.2:45809/ using document root <none> and password file <none>
I20260619 17:00:23.365756 2015 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:23.365881 2015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:23.366238 2015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:23.369562 2015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "6f98d9e0498c413383d8ed877d5bf724"
format_stamp: "Formatted at 2026-06-19 17:00:23 on dist-test-slave-3ffd"
server_key: "5ea8794cbc2fd6311916bc9cc34ce405"
server_key_iv: "2e22211435c1a913d675aa583ae099bf"
server_key_version: "encryptionkey@0"
I20260619 17:00:23.370501 2015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "6f98d9e0498c413383d8ed877d5bf724"
format_stamp: "Formatted at 2026-06-19 17:00:23 on dist-test-slave-3ffd"
server_key: "5ea8794cbc2fd6311916bc9cc34ce405"
server_key_iv: "2e22211435c1a913d675aa583ae099bf"
server_key_version: "encryptionkey@0"
I20260619 17:00:23.376637 2015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 17:00:23.381022 2029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:23.382717 2015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 17:00:23.382994 2015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "6f98d9e0498c413383d8ed877d5bf724"
format_stamp: "Formatted at 2026-06-19 17:00:23 on dist-test-slave-3ffd"
server_key: "5ea8794cbc2fd6311916bc9cc34ce405"
server_key_iv: "2e22211435c1a913d675aa583ae099bf"
server_key_version: "encryptionkey@0"
I20260619 17:00:23.383262 2015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:23.404085 2015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:23.405185 2015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:23.405529 2015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:23.406703 2015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:23.408434 2015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:23.408574 2015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:23.408717 2015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:23.408820 2015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:23.446102 2015 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:34601
I20260619 17:00:23.446116 2141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:34601 every 8 connection(s)
I20260619 17:00:23.447598 2015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 17:00:23.457690 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 2015
I20260619 17:00:23.457885 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 17:00:23.458575 16576 external_mini_cluster.cc:1428] Setting key 748253669605fc1b333c96b6e966ce2f
I20260619 17:00:23.461824 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:23.450539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:39397 (local address 127.16.48.62:36509)
0619 17:00:23.450733 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:23.450737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:23.452302 (+ 1565us) server_negotiation.cc:407] Connection header received
0619 17:00:23.454199 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:23.454205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:23.454307 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:23.454432 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:23.455596 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.456407 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.458522 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.458994 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.459172 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:23.460882 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:23.460889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:23.460893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:23.460916 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:23.460960 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:23.460977 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:23.461077 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:23.461300 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 17:00:23.461405 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 17:00:23.461962 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36509
--builtin_ntp_servers=127.16.48.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 17:00:23.462852 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:23.451085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36509 (local address 127.16.48.2:39397)
0619 17:00:23.452062 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:23.452099 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:23.453885 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:23.454673 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:23.454682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:23.454779 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:23.455391 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:23.455434 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.456569 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.456574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:23.458358 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:23.458376 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.459287 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.459298 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:23.459608 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:23.460472 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 17:00:23.460476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:23.460539 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 17:00:23.460662 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:23.461084 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:23.461093 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:23.461100 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 17:00:23.461256 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 17:00:23.461611 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":318,"threads_started":1}
I20260619 17:00:23.464404 2142 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36509
I20260619 17:00:23.464744 2142 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:23.465541 2142 heartbeater.cc:511] Master 127.16.48.62:36509 requested a full tablet report, sending...
I20260619 17:00:23.467034 1825 ts_manager.cc:194] Registered new tserver with Master: 6f98d9e0498c413383d8ed877d5bf724 (127.16.48.2:34601)
I20260619 17:00:23.467878 1825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.2:39397
W20260619 17:00:23.812700 2145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:23.813030 2145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:23.813206 2145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:23.820859 2145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:23.820977 2145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:23.821173 2145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 17:00:23.830737 2145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:23.832231 2145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:23.834414 2145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:23.848917 2150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:23.848953 2151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:23.849006 2153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:23.849820 2145 server_base.cc:1034] running on GCE node
I20260619 17:00:23.850535 2145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:23.851616 2145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:23.852872 2145 hybrid_clock.cc:648] HybridClock initialized: now 1781888423852831 us; error 33 us; skew 500 ppm
I20260619 17:00:23.853242 2145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:23.855705 2145 webserver.cc:460] Webserver started at http://127.16.48.3:33077/ using document root <none> and password file <none>
I20260619 17:00:23.856575 2145 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:23.856715 2145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:23.857074 2145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:23.860962 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "75914aff1fb04647a6fababd11931459"
format_stamp: "Formatted at 2026-06-19 17:00:23 on dist-test-slave-3ffd"
server_key: "9e6f731aa877965bd0abe5c6ae2e6d3d"
server_key_iv: "cf56559a049e9190e5bea62eb2f852fe"
server_key_version: "encryptionkey@0"
I20260619 17:00:23.862015 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "75914aff1fb04647a6fababd11931459"
format_stamp: "Formatted at 2026-06-19 17:00:23 on dist-test-slave-3ffd"
server_key: "9e6f731aa877965bd0abe5c6ae2e6d3d"
server_key_iv: "cf56559a049e9190e5bea62eb2f852fe"
server_key_version: "encryptionkey@0"
I20260619 17:00:23.868659 2145 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 17:00:23.873133 2159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:23.874719 2145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 17:00:23.874930 2145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "75914aff1fb04647a6fababd11931459"
format_stamp: "Formatted at 2026-06-19 17:00:23 on dist-test-slave-3ffd"
server_key: "9e6f731aa877965bd0abe5c6ae2e6d3d"
server_key_iv: "cf56559a049e9190e5bea62eb2f852fe"
server_key_version: "encryptionkey@0"
I20260619 17:00:23.875181 2145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:23.907109 2145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:23.908293 2145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:23.908669 2145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:23.910130 2145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:23.912174 2145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:23.912322 2145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:23.912492 2145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:23.912587 2145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:23.947738 2145 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:40955
I20260619 17:00:23.947800 2271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:40955 every 8 connection(s)
I20260619 17:00:23.949213 2145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 17:00:23.950276 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 2145
I20260619 17:00:23.950515 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 17:00:23.951092 16576 external_mini_cluster.cc:1428] Setting key b4455930825dbc71fa81cfec84044717
I20260619 17:00:23.965708 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:23.953303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:37817 (local address 127.16.48.62:36509)
0619 17:00:23.953494 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:23.953500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:23.955282 (+ 1782us) server_negotiation.cc:407] Connection header received
0619 17:00:23.958303 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:23.958311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:23.958406 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:23.958503 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:23.959903 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.960759 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.962561 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.963000 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.963107 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:23.964773 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:23.964782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:23.964787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:23.964832 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:23.964867 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:23.964881 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:23.964964 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:23.965192 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 17:00:23.965287 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 17:00:23.966951 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:23.953765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36509 (local address 127.16.48.3:37817)
0619 17:00:23.955074 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:23.955134 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:23.957788 (+ 2654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:23.958836 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:23.958852 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:23.958997 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:23.959697 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:23.959717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.960939 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.960944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:23.962344 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:23.962359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.963261 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.963267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:23.963536 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:23.964380 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 17:00:23.964383 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:23.964433 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 17:00:23.964522 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:23.964969 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:23.964977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:23.964984 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 17:00:23.965154 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 17:00:23.965532 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":422,"threads_started":1}
I20260619 17:00:23.968611 2272 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36509
I20260619 17:00:23.968945 2272 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:23.969753 2272 heartbeater.cc:511] Master 127.16.48.62:36509 requested a full tablet report, sending...
I20260619 17:00:23.971264 1824 ts_manager.cc:194] Registered new tserver with Master: 75914aff1fb04647a6fababd11931459 (127.16.48.3:40955)
I20260619 17:00:23.972259 1824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.3:37817
I20260619 17:00:23.978320 2011 heartbeater.cc:503] Master 127.16.48.62:36509 was elected leader, sending a full tablet report...
I20260619 17:00:23.981186 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 17:00:23.991535 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:23.985224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.16.48.62:36509)
0619 17:00:23.985486 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:23.985496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:23.985834 (+ 338us) server_negotiation.cc:407] Connection header received
0619 17:00:23.985963 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:23.985967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:23.986045 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:23.986138 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:23.986574 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.987291 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.988933 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:23.989390 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:23.989556 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:23.990737 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:23.990742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:23.990745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:23.990767 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:23.990801 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:23.990813 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:23.990913 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:23.991061 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 17:00:23.991171 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 17:00:23.996932 1824 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 17:00:24.004292 1824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 17:00:24.035183 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.017839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:34601 (local address 127.0.0.1:41106)
0619 17:00:24.019085 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.019122 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:24.019316 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:24.020420 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:24.020425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:24.020447 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:24.022435 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.022456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.026908 (+ 4452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.026917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.032169 (+ 5252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.032190 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.034050 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.034060 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.034370 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.034470 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 17:00:24.034615 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 17:00:24.034843 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":794,"threads_started":1}
I20260619 17:00:24.036327 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.017941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.16.48.2:34601)
0619 17:00:24.018605 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:24.018615 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:24.019242 (+ 627us) server_negotiation.cc:407] Connection header received
0619 17:00:24.019563 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:24.019572 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:24.019879 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:24.020094 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:24.022734 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.026677 (+ 3943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.032447 (+ 5770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.033831 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.033976 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.034283 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:24.035712 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0619 17:00:24.035910 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":332,"threads_started":1}
I20260619 17:00:24.042757 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 9005be317e11469fbc29222efe8e581e (DEFAULT_TABLE table=test-table [id=2fae630f51c44ab0b5952433acdc5b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:24.044513 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9005be317e11469fbc29222efe8e581e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:24.044935 2286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.019515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45779 (local address 127.0.0.1:57468)
0619 17:00:24.020101 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.020170 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:24.020284 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:24.033668 (+ 13384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:24.033675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:24.033703 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:24.034128 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.034144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.037821 (+ 3677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.037827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.040526 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.040544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.043773 (+ 3229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.043780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.044328 (+ 548us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.044364 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 17:00:24.044465 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 17:00:24.044614 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":285,"threads_started":1}
I20260619 17:00:24.045886 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.019879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.16.48.1:45779)
0619 17:00:24.033126 (+ 13247us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:24.033135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:24.033164 (+ 29us) server_negotiation.cc:407] Connection header received
0619 17:00:24.033278 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:24.033284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:24.033451 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:24.033683 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:24.034379 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.036443 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.040773 (+ 4330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.043216 (+ 2443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.043356 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.043634 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:24.045344 (+ 1710us) server_negotiation.cc:299] Negotiation successful
0619 17:00:24.045543 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13044,"thread_start_us":13128,"threads_started":1}
I20260619 17:00:24.048388 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.020209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:40955 (local address 127.0.0.1:47918)
0619 17:00:24.032039 (+ 11830us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.032068 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:24.032887 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:24.033967 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:24.033972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:24.033999 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:24.034474 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.034485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.036422 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.036429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.038101 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.038120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.040274 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.040301 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.040597 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.040676 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 17:00:24.040823 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 17:00:24.041034 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11660,"thread_start_us":11494,"threads_started":1}
I20260619 17:00:24.050077 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.020880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.16.48.3:40955)
0619 17:00:24.032546 (+ 11666us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:24.032560 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:24.032609 (+ 49us) server_negotiation.cc:407] Connection header received
0619 17:00:24.033128 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:24.033140 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:24.033452 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:24.033671 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:24.034670 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.036199 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.038382 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.039910 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.040084 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.049578 (+ 9494us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:24.049652 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 17:00:24.049790 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11408,"thread_start_us":11253,"threads_started":1}
I20260619 17:00:24.052625 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 9005be317e11469fbc29222efe8e581e (DEFAULT_TABLE table=test-table [id=2fae630f51c44ab0b5952433acdc5b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:24.053700 2207 tablet_service.cc:1467] Processing CreateTablet for tablet 9005be317e11469fbc29222efe8e581e (DEFAULT_TABLE table=test-table [id=2fae630f51c44ab0b5952433acdc5b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:24.054328 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9005be317e11469fbc29222efe8e581e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:24.054935 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9005be317e11469fbc29222efe8e581e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:24.063021 2290 tablet_bootstrap.cc:492] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: Bootstrap starting.
I20260619 17:00:24.067538 2290 tablet_bootstrap.cc:654] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:24.068931 2292 tablet_bootstrap.cc:492] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: Bootstrap starting.
I20260619 17:00:24.068993 2290 log.cc:824] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:24.070792 2291 tablet_bootstrap.cc:492] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: Bootstrap starting.
I20260619 17:00:24.072870 2290 tablet_bootstrap.cc:492] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: No bootstrap required, opened a new log
I20260619 17:00:24.073313 2290 ts_tablet_manager.cc:1397] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 17:00:24.073374 2292 tablet_bootstrap.cc:654] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:24.074507 2292 log.cc:824] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:24.075431 2291 tablet_bootstrap.cc:654] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:24.076752 2291 log.cc:824] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:24.078001 2292 tablet_bootstrap.cc:492] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: No bootstrap required, opened a new log
I20260619 17:00:24.078305 2292 ts_tablet_manager.cc:1397] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260619 17:00:24.080260 2291 tablet_bootstrap.cc:492] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: No bootstrap required, opened a new log
I20260619 17:00:24.080545 2291 ts_tablet_manager.cc:1397] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 17:00:24.084484 2290 raft_consensus.cc:348] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } }
I20260619 17:00:24.085026 2290 raft_consensus.cc:374] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:24.085157 2290 raft_consensus.cc:729] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f98d9e0498c413383d8ed877d5bf724, State: Initialized, Role: FOLLOWER
I20260619 17:00:24.085876 2290 consensus_queue.cc:260] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } }
I20260619 17:00:24.087127 2292 raft_consensus.cc:348] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } }
I20260619 17:00:24.087524 2292 raft_consensus.cc:374] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:24.087661 2292 raft_consensus.cc:729] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75914aff1fb04647a6fababd11931459, State: Initialized, Role: FOLLOWER
I20260619 17:00:24.088375 2292 consensus_queue.cc:260] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } }
I20260619 17:00:24.089526 2272 heartbeater.cc:503] Master 127.16.48.62:36509 was elected leader, sending a full tablet report...
I20260619 17:00:24.091588 2292 ts_tablet_manager.cc:1428] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 17:00:24.092605 2142 heartbeater.cc:503] Master 127.16.48.62:36509 was elected leader, sending a full tablet report...
I20260619 17:00:24.093253 2290 ts_tablet_manager.cc:1428] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260619 17:00:24.093703 2291 raft_consensus.cc:348] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } }
I20260619 17:00:24.094251 2291 raft_consensus.cc:374] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:24.094373 2291 raft_consensus.cc:729] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9499f3a21b4603aec33ffae8ddd21a, State: Initialized, Role: FOLLOWER
I20260619 17:00:24.095023 2291 consensus_queue.cc:260] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } }
I20260619 17:00:24.097471 2291 ts_tablet_manager.cc:1428] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260619 17:00:24.179410 2298 raft_consensus.cc:482] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 17:00:24.179777 2298 raft_consensus.cc:504] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } }
I20260619 17:00:24.181777 2298 leader_election.cc:288] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f98d9e0498c413383d8ed877d5bf724 (127.16.48.2:34601), 75914aff1fb04647a6fababd11931459 (127.16.48.3:40955)
I20260619 17:00:24.189440 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.182395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:46475 (local address 127.16.48.2:34601)
0619 17:00:24.182674 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:24.182679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:24.183068 (+ 389us) server_negotiation.cc:407] Connection header received
0619 17:00:24.183167 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:24.183172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:24.183266 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:24.183380 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:24.184359 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.185074 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.187329 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.188439 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.188599 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.188700 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:24.189018 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 17:00:24.189151 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 17:00:24.189507 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.182139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:34601 (local address 127.16.48.1:46475)
0619 17:00:24.182794 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.182816 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:24.182966 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:24.183650 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:24.183660 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:24.183703 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:24.184182 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.184200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.185232 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.185241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.187048 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.187069 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.188641 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.188648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.188841 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.188866 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 17:00:24.188978 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 17:00:24.189129 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":341,"threads_started":1}
I20260619 17:00:24.189930 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.183224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:33015 (local address 127.16.48.3:40955)
0619 17:00:24.183424 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:24.183431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:24.183930 (+ 499us) server_negotiation.cc:407] Connection header received
0619 17:00:24.184175 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:24.184181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:24.184250 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:24.184347 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:24.184877 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.185620 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.187983 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.188994 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.189153 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.189265 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:24.189555 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 17:00:24.189689 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 17:00:24.189993 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.182959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:40955 (local address 127.16.48.1:33015)
0619 17:00:24.183812 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.183827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:24.183990 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:24.184364 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:24.184369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:24.184392 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:24.184719 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.184727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.185767 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.185772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.187764 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:24.187777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.189210 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.189217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:24.189390 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.189414 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 17:00:24.189534 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 17:00:24.189677 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":416,"threads_started":1}
I20260619 17:00:24.191136 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9005be317e11469fbc29222efe8e581e" candidate_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f98d9e0498c413383d8ed877d5bf724" is_pre_election: true
I20260619 17:00:24.191152 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9005be317e11469fbc29222efe8e581e" candidate_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75914aff1fb04647a6fababd11931459" is_pre_election: true
I20260619 17:00:24.191690 2097 raft_consensus.cc:2454] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d9499f3a21b4603aec33ffae8ddd21a in term 0.
I20260619 17:00:24.191725 2227 raft_consensus.cc:2454] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d9499f3a21b4603aec33ffae8ddd21a in term 0.
I20260619 17:00:24.192502 1900 leader_election.cc:302] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9499f3a21b4603aec33ffae8ddd21a, 75914aff1fb04647a6fababd11931459; no voters:
I20260619 17:00:24.192948 2298 raft_consensus.cc:2781] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 17:00:24.193128 2298 raft_consensus.cc:482] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 17:00:24.193281 2298 raft_consensus.cc:3037] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:24.195618 2298 raft_consensus.cc:504] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } }
I20260619 17:00:24.196412 2298 leader_election.cc:288] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [CANDIDATE]: Term 1 election: Requested vote from peers 6f98d9e0498c413383d8ed877d5bf724 (127.16.48.2:34601), 75914aff1fb04647a6fababd11931459 (127.16.48.3:40955)
I20260619 17:00:24.197131 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9005be317e11469fbc29222efe8e581e" candidate_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f98d9e0498c413383d8ed877d5bf724"
I20260619 17:00:24.197232 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9005be317e11469fbc29222efe8e581e" candidate_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75914aff1fb04647a6fababd11931459"
I20260619 17:00:24.197402 2097 raft_consensus.cc:3037] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:24.197495 2227 raft_consensus.cc:3037] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:24.199955 2227 raft_consensus.cc:2454] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d9499f3a21b4603aec33ffae8ddd21a in term 1.
W20260619 17:00:24.200008 2143 tablet.cc:2369] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:24.200029 2097 raft_consensus.cc:2454] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d9499f3a21b4603aec33ffae8ddd21a in term 1.
I20260619 17:00:24.200606 1900 leader_election.cc:302] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9499f3a21b4603aec33ffae8ddd21a, 6f98d9e0498c413383d8ed877d5bf724; no voters:
I20260619 17:00:24.200954 2298 raft_consensus.cc:2781] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:24.201516 2298 raft_consensus.cc:686] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 1 LEADER]: Becoming Leader. State: Replica: 2d9499f3a21b4603aec33ffae8ddd21a, State: Running, Role: LEADER
I20260619 17:00:24.202056 2298 consensus_queue.cc:237] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } }
W20260619 17:00:24.202867 2273 tablet.cc:2369] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 17:00:24.205283 2012 tablet.cc:2369] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:24.207743 1825 catalog_manager.cc:5495] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d9499f3a21b4603aec33ffae8ddd21a (127.16.48.1). New cstate: current_term: 1 leader_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d9499f3a21b4603aec33ffae8ddd21a" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 } health_report { overall_health: UNKNOWN } } }
I20260619 17:00:24.227052 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.221509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.16.48.1:45779)
0619 17:00:24.221739 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:24.221744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:24.221760 (+ 16us) server_negotiation.cc:407] Connection header received
0619 17:00:24.221852 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:24.221856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:24.221952 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:24.222121 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:00:24.222503 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.223237 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.224783 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.225227 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.225372 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.225904 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:00:24.226462 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:00:24.226631 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:24.226690 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 17:00:24.226779 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 17:00:24.235889 2097 raft_consensus.cc:1261] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [term 1 FOLLOWER]: Refusing update from remote peer 2d9499f3a21b4603aec33ffae8ddd21a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:24.235891 2227 raft_consensus.cc:1261] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [term 1 FOLLOWER]: Refusing update from remote peer 2d9499f3a21b4603aec33ffae8ddd21a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:24.236950 2301 consensus_queue.cc:1035] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f98d9e0498c413383d8ed877d5bf724" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 34601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:24.237363 2298 consensus_queue.cc:1035] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75914aff1fb04647a6fababd11931459" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:24.267417 1825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:24.267711 1825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:24.271723 1825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e4b66f2e4084f1cb7431ffc6165ec1e: Sending DeleteTablet for 3 replicas of tablet 9005be317e11469fbc29222efe8e581e
I20260619 17:00:24.273094 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet 9005be317e11469fbc29222efe8e581e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:24 UTC) from {username='slave'} at 127.0.0.1:41106
I20260619 17:00:24.273240 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 9005be317e11469fbc29222efe8e581e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:24 UTC) from {username='slave'} at 127.0.0.1:57468
I20260619 17:00:24.273222 2207 tablet_service.cc:1514] Processing DeleteTablet for tablet 9005be317e11469fbc29222efe8e581e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:24 UTC) from {username='slave'} at 127.0.0.1:47918
I20260619 17:00:24.275355 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 1884
I20260619 17:00:24.289580 2316 tablet_replica.cc:330] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: stopping tablet replica
I20260619 17:00:24.290300 2316 raft_consensus.cc:2229] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:24.291005 2316 raft_consensus.cc:2258] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:24.295085 2315 tablet_replica.cc:330] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: stopping tablet replica
I20260619 17:00:24.295652 2315 raft_consensus.cc:2229] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 1 LEADER]: Raft consensus shutting down.
I20260619 17:00:24.296414 2314 tablet_replica.cc:330] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: stopping tablet replica
I20260619 17:00:24.296495 2315 raft_consensus.cc:2258] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:24.297508 2316 ts_tablet_manager.cc:1905] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:24.297483 2314 raft_consensus.cc:2229] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:24.298020 2314 raft_consensus.cc:2258] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:24.299799 2315 ts_tablet_manager.cc:1905] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:24.304615 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.276354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.16.48.1:45779)
0619 17:00:24.276549 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:24.276555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:24.293372 (+ 16817us) server_negotiation.cc:407] Connection header received
0619 17:00:24.294023 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:24.294038 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:24.294227 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:24.294339 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:24.296708 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.297936 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.300161 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.300886 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.301006 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.302630 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:24.302650 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:24.302673 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:24.303538 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:24.303605 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:24.303640 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:24.303725 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:24.304022 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 17:00:24.304178 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 17:00:24.305899 2316 ts_tablet_manager.cc:1918] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:24.306131 2316 log.cc:1196] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/9005be317e11469fbc29222efe8e581e
I20260619 17:00:24.306269 2006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:24.306757 2316 ts_tablet_manager.cc:1939] T 9005be317e11469fbc29222efe8e581e P 75914aff1fb04647a6fababd11931459: Deleting consensus metadata
I20260619 17:00:24.308697 2314 ts_tablet_manager.cc:1905] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:24.308856 1810 catalog_manager.cc:4841] TS 75914aff1fb04647a6fababd11931459 (127.16.48.3:40955): tablet 9005be317e11469fbc29222efe8e581e (table test-table [id=2fae630f51c44ab0b5952433acdc5b03]) successfully deleted
I20260619 17:00:24.315572 2314 ts_tablet_manager.cc:1918] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:24.315763 2314 log.cc:1196] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/9005be317e11469fbc29222efe8e581e
I20260619 17:00:24.316308 2314 ts_tablet_manager.cc:1939] T 9005be317e11469fbc29222efe8e581e P 6f98d9e0498c413383d8ed877d5bf724: Deleting consensus metadata
I20260619 17:00:24.318259 1810 catalog_manager.cc:4841] TS 6f98d9e0498c413383d8ed877d5bf724 (127.16.48.2:34601): tablet 9005be317e11469fbc29222efe8e581e (table test-table [id=2fae630f51c44ab0b5952433acdc5b03]) successfully deleted
I20260619 17:00:24.407651 2315 ts_tablet_manager.cc:1918] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:24.407977 2315 log.cc:1196] T 9005be317e11469fbc29222efe8e581e P 2d9499f3a21b4603aec33ffae8ddd21a: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/9005be317e11469fbc29222efe8e581e
I20260619 17:00:24.408459 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 1884
I20260619 17:00:24.427873 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 2015
W20260619 17:00:24.428054 1810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.48.1:45779 (error 108)
I20260619 17:00:24.430474 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.430028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45779 (local address 127.0.0.1:57482)
0619 17:00:24.430221 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.430330 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 17:00:24.430724 1810 catalog_manager.cc:4584] TS 2d9499f3a21b4603aec33ffae8ddd21a (127.16.48.1:45779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9005be317e11469fbc29222efe8e581e: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45779: connect: Connection refused (error 111)
I20260619 17:00:24.435338 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.428911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.16.48.2:34601)
0619 17:00:24.429150 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:24.429155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:24.429178 (+ 23us) server_negotiation.cc:407] Connection header received
0619 17:00:24.429318 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:24.429323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:24.429419 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:24.429546 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:24.429981 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.430852 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.432319 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.432750 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.432853 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.433881 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:24.433890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:24.433906 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:24.434527 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:24.434576 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:24.434596 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:24.434689 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:24.434832 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 17:00:24.434933 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 17:00:24.436275 2137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:24.484632 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.484000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45779 (local address 127.0.0.1:57492)
0619 17:00:24.484297 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.484416 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 17:00:24.485917 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.485459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45779 (local address 127.0.0.1:57502)
0619 17:00:24.485645 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.485744 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 17:00:24.529011 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 2015
I20260619 17:00:24.546721 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.546101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45779 (local address 127.0.0.1:57504)
0619 17:00:24.546441 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.546552 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260619 17:00:24.547370 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 2145
I20260619 17:00:24.548022 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.547552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45779 (local address 127.0.0.1:57514)
0619 17:00:24.547720 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.547842 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 17:00:24.554903 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.548277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.16.48.3:40955)
0619 17:00:24.548470 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:24.548474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:24.548493 (+ 19us) server_negotiation.cc:407] Connection header received
0619 17:00:24.548634 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:24.548638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:24.548734 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:24.548823 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:24.549196 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.550132 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.551482 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:24.551913 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:24.552017 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:24.553410 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:24.553421 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:24.553438 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:24.554075 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:24.554129 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:24.554157 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:24.554256 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:24.554418 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 17:00:24.554522 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 17:00:24.555637 2267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:24.628952 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.628366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45779 (local address 127.0.0.1:57522)
0619 17:00:24.628610 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.628724 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 17:00:24.630229 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:24.629722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45779 (local address 127.0.0.1:57532)
0619 17:00:24.629973 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:24.630076 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 17:00:24.645702 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 2145
I20260619 17:00:24.662830 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 1794
I20260619 17:00:24.664182 1855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:24.785040 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 1794
2026-06-19T17:00:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3015 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:24 dist-test-slave-3ffd krb5kdc[2325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:24 dist-test-slave-3ffd krb5kdc[2325](info): set up 2 sockets
Jun 19 17:00:24 dist-test-slave-3ffd krb5kdc[2325](info): commencing operation
krb5kdc: starting...
W20260619 17:00:25.924741 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:26 dist-test-slave-3ffd krb5kdc[2325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:00:26.204665 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:43495
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:43495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:26.560312 2338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:26.560690 2338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:26.560864 2338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:26.568732 2338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:26.568909 2338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:26.569026 2338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:26.569135 2338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:26.569222 2338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:26.579859 2338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:43495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:43495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:26.581408 2338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:26.582849 2338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:26.596362 2346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:26.596439 2344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:26.596335 2343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:26.597007 2338 server_base.cc:1034] running on GCE node
I20260619 17:00:26.598062 2338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:26.599828 2338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:26.601267 2338 hybrid_clock.cc:648] HybridClock initialized: now 1781888426601210 us; error 50 us; skew 500 ppm
I20260619 17:00:26.601722 2338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:26 dist-test-slave-3ffd krb5kdc[2325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:26.612003 2338 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:00:26.613994 2338 webserver.cc:460] Webserver started at http://127.16.48.62:35209/ using document root <none> and password file <none>
I20260619 17:00:26.614790 2338 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:26.614939 2338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:26.615309 2338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:26.618790 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "91c0d922a9f04af9bdca8a7b2ca818bc"
format_stamp: "Formatted at 2026-06-19 17:00:26 on dist-test-slave-3ffd"
server_key: "f8b9174e84c3a5ad527cbe665ce763af"
server_key_iv: "5b8694413cee37b81ab12ce36e4c0dfe"
server_key_version: "encryptionkey@0"
I20260619 17:00:26.619689 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "91c0d922a9f04af9bdca8a7b2ca818bc"
format_stamp: "Formatted at 2026-06-19 17:00:26 on dist-test-slave-3ffd"
server_key: "f8b9174e84c3a5ad527cbe665ce763af"
server_key_iv: "5b8694413cee37b81ab12ce36e4c0dfe"
server_key_version: "encryptionkey@0"
I20260619 17:00:26.625846 2338 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 17:00:26.629999 2353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:26.631635 2338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 17:00:26.631860 2338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "91c0d922a9f04af9bdca8a7b2ca818bc"
format_stamp: "Formatted at 2026-06-19 17:00:26 on dist-test-slave-3ffd"
server_key: "f8b9174e84c3a5ad527cbe665ce763af"
server_key_iv: "5b8694413cee37b81ab12ce36e4c0dfe"
server_key_version: "encryptionkey@0"
I20260619 17:00:26.632110 2338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:26.643949 2338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:26.648947 2338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:26.649322 2338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:26.669889 2338 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:43495
I20260619 17:00:26.669919 2404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:43495 every 8 connection(s)
I20260619 17:00:26.671576 2338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:26.675889 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:26.681843 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 2338
I20260619 17:00:26.682060 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:26.682507 16576 external_mini_cluster.cc:1428] Setting key d2933d64aee98f877856944c76cd4985
I20260619 17:00:26.689417 2405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc: Bootstrap starting.
I20260619 17:00:26.693780 2405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:26.695000 2405 log.cc:824] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:26.699663 2405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc: No bootstrap required, opened a new log
Jun 19 17:00:26 dist-test-slave-3ffd krb5kdc[2325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:26.711252 2405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0d922a9f04af9bdca8a7b2ca818bc" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43495 } }
I20260619 17:00:26.711755 2405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:26.711884 2405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c0d922a9f04af9bdca8a7b2ca818bc, State: Initialized, Role: FOLLOWER
I20260619 17:00:26.712733 2405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0d922a9f04af9bdca8a7b2ca818bc" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43495 } }
I20260619 17:00:26.713043 2405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:26.713212 2405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:26.713406 2405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:26.713806 2408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:26.684669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.16.48.62:43495)
0619 17:00:26.685667 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:26.685685 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:26.685745 (+ 60us) server_negotiation.cc:407] Connection header received
0619 17:00:26.688353 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:26.688392 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:26.688997 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:26.689512 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:26.691125 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:26.693280 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:26.694968 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:26.696333 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:26.696600 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:26.703281 (+ 6681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:26.703301 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:26.703329 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:26.703393 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:26.708648 (+ 5255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:26.709618 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:26.709633 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:26.709644 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:26.709893 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:26.710363 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:26.710368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:26.710372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:26.711020 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:26.711395 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:26.711930 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 17:00:26.712494 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":417,"threads_started":1}
I20260619 17:00:26.716760 2405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0d922a9f04af9bdca8a7b2ca818bc" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43495 } }
I20260619 17:00:26.717437 2405 leader_election.cc:302] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91c0d922a9f04af9bdca8a7b2ca818bc; no voters:
I20260619 17:00:26.718179 2405 leader_election.cc:288] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:26.718319 2410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:26.719074 2410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [term 1 LEADER]: Becoming Leader. State: Replica: 91c0d922a9f04af9bdca8a7b2ca818bc, State: Running, Role: LEADER
I20260619 17:00:26.719669 2410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0d922a9f04af9bdca8a7b2ca818bc" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43495 } }
I20260619 17:00:26.720510 2405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:26.722844 2411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91c0d922a9f04af9bdca8a7b2ca818bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0d922a9f04af9bdca8a7b2ca818bc" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43495 } } }
I20260619 17:00:26.722908 2412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c0d922a9f04af9bdca8a7b2ca818bc. Latest consensus state: current_term: 1 leader_uuid: "91c0d922a9f04af9bdca8a7b2ca818bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0d922a9f04af9bdca8a7b2ca818bc" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 43495 } } }
I20260619 17:00:26.723332 2411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:26.723368 2412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:26.729449 2416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:26.735551 2416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:26.745664 2416 catalog_manager.cc:1347] Generated new cluster ID: c52590b5c9cd4c3aa401c107fbfd283a
I20260619 17:00:26.745805 2416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:26.769451 2416 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:26.771584 2416 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:26.781550 2416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc: Generated new TSK 0
I20260619 17:00:26.782739 2416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 17:00:26.864064 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43495
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:27.241643 2431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:27.241951 2431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:27.242113 2431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:27.249941 2431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:27.250083 2431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:27.250288 2431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 17:00:27.261042 2431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:27.262467 2431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:27.263918 2431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:27.278743 2436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:27.278683 2437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:27.278668 2439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:27.279618 2431 server_base.cc:1034] running on GCE node
I20260619 17:00:27.280381 2431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:27.281466 2431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:27.282743 2431 hybrid_clock.cc:648] HybridClock initialized: now 1781888427282672 us; error 71 us; skew 500 ppm
I20260619 17:00:27.283124 2431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:27 dist-test-slave-3ffd krb5kdc[2325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888427, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:27.286764 2431 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 17:00:27.288437 2431 webserver.cc:460] Webserver started at http://127.16.48.1:37133/ using document root <none> and password file <none>
I20260619 17:00:27.289218 2431 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:27.289330 2431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:27.289609 2431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:27.292909 2431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "de733238abed41c384dda03d2df56612"
format_stamp: "Formatted at 2026-06-19 17:00:27 on dist-test-slave-3ffd"
server_key: "d88ec8591e1f8f631786d20001d27230"
server_key_iv: "b94ad4a2bf4b001407289a554b342c00"
server_key_version: "encryptionkey@0"
I20260619 17:00:27.293768 2431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "de733238abed41c384dda03d2df56612"
format_stamp: "Formatted at 2026-06-19 17:00:27 on dist-test-slave-3ffd"
server_key: "d88ec8591e1f8f631786d20001d27230"
server_key_iv: "b94ad4a2bf4b001407289a554b342c00"
server_key_version: "encryptionkey@0"
I20260619 17:00:27.299724 2431 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260619 17:00:27.303855 2446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:27.305595 2431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 17:00:27.305814 2431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "de733238abed41c384dda03d2df56612"
format_stamp: "Formatted at 2026-06-19 17:00:27 on dist-test-slave-3ffd"
server_key: "d88ec8591e1f8f631786d20001d27230"
server_key_iv: "b94ad4a2bf4b001407289a554b342c00"
server_key_version: "encryptionkey@0"
I20260619 17:00:27.306068 2431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:27.326046 2431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:27.330761 2431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:27.331110 2431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:27.332252 2431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:27.333789 2431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:27.333869 2431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:27.334059 2431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:27.334120 2431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:27.370488 2431 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:39039
I20260619 17:00:27.370528 2558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:39039 every 8 connection(s)
I20260619 17:00:27.371974 2431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 17:00:27.380975 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 2431
I20260619 17:00:27.381211 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 17:00:27.381687 16576 external_mini_cluster.cc:1428] Setting key f2a4e2733435a5493dacf82a2bf8581a
Jun 19 17:00:27 dist-test-slave-3ffd krb5kdc[2325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888427, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:27.394418 2561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:27.374872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:51049 (local address 127.16.48.62:43495)
0619 17:00:27.375390 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:27.375395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:27.376433 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 17:00:27.378483 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:27.378489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:27.378607 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:27.378739 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:27.380574 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:27.381454 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:27.383436 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:27.384006 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:27.384197 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:27.388105 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:27.388111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:27.388160 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:27.388210 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:27.390920 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:27.391846 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:27.391855 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:27.391863 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:27.391992 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:27.392497 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:27.392501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:27.392504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:27.392805 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:27.392973 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:27.393943 (+ 970us) server_negotiation.cc:299] Negotiation successful
0619 17:00:27.394159 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":276,"threads_started":1}
I20260619 17:00:27.396239 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:27.375380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43495 (local address 127.16.48.1:51049)
0619 17:00:27.376252 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:27.376299 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:27.378138 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:27.379035 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:27.379047 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:27.379652 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:27.380368 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:27.380390 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:27.381654 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:27.381658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:27.383264 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:27.383280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:27.384322 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:27.384330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:27.384605 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:27.385390 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:00:27.385421 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:27.387796 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:27.391139 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:27.391150 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:27.391171 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:27.391691 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:27.392142 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:27.392147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:27.392151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:27.392356 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:27.392947 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:27.392956 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:27.393459 (+ 503us) client_negotiation.cc:769] Sending connection context
0619 17:00:27.393801 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 17:00:27.394438 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":311,"threads_started":1}
I20260619 17:00:27.398118 2559 heartbeater.cc:347] Connected to a master server at 127.16.48.62:43495
I20260619 17:00:27.398475 2559 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:27.399303 2559 heartbeater.cc:511] Master 127.16.48.62:43495 requested a full tablet report, sending...
I20260619 17:00:27.401686 2370 ts_manager.cc:194] Registered new tserver with Master: de733238abed41c384dda03d2df56612 (127.16.48.1:39039)
I20260619 17:00:27.403249 2370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:51049
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 17:00:27.443540 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43495
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:27.793658 2565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:27.793968 2565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:27.794076 2565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:27.801729 2565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:27.801868 2565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:27.801987 2565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 17:00:27.812776 2565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:27.814170 2565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:27.815528 2565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:27.829792 2573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:27.829821 2570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:27.829856 2571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:27.831130 2565 server_base.cc:1034] running on GCE node
I20260619 17:00:27.831807 2565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:27.832926 2565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:27.834183 2565 hybrid_clock.cc:648] HybridClock initialized: now 1781888427834133 us; error 47 us; skew 500 ppm
I20260619 17:00:27.834548 2565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:27 dist-test-slave-3ffd krb5kdc[2325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888427, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:27.838218 2565 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 17:00:27.839936 2565 webserver.cc:460] Webserver started at http://127.16.48.2:42257/ using document root <none> and password file <none>
I20260619 17:00:27.840794 2565 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:27.840961 2565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:27.841313 2565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:27.844581 2565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "c3a45d45137e4b78999146bc9231285c"
format_stamp: "Formatted at 2026-06-19 17:00:27 on dist-test-slave-3ffd"
server_key: "36059c720e63d5464635b1c3e631d4c7"
server_key_iv: "434824c28dbeb5764d0ec899fc8f8125"
server_key_version: "encryptionkey@0"
I20260619 17:00:27.845474 2565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "c3a45d45137e4b78999146bc9231285c"
format_stamp: "Formatted at 2026-06-19 17:00:27 on dist-test-slave-3ffd"
server_key: "36059c720e63d5464635b1c3e631d4c7"
server_key_iv: "434824c28dbeb5764d0ec899fc8f8125"
server_key_version: "encryptionkey@0"
I20260619 17:00:27.851377 2565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 17:00:27.855683 2580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:27.857340 2565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 17:00:27.857566 2565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "c3a45d45137e4b78999146bc9231285c"
format_stamp: "Formatted at 2026-06-19 17:00:27 on dist-test-slave-3ffd"
server_key: "36059c720e63d5464635b1c3e631d4c7"
server_key_iv: "434824c28dbeb5764d0ec899fc8f8125"
server_key_version: "encryptionkey@0"
I20260619 17:00:27.857817 2565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:27.872526 2565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:27.877610 2565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:27.877952 2565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:27.879112 2565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:27.880668 2565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:27.880774 2565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:27.880926 2565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:27.881000 2565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:27.916436 2565 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:43965
I20260619 17:00:27.916478 2692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:43965 every 8 connection(s)
I20260619 17:00:27.917899 2565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 17:00:27.920086 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 2565
I20260619 17:00:27.920352 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 17:00:27.920991 16576 external_mini_cluster.cc:1428] Setting key 1c2fb6582449ff6c6c1f9be9cc1bfeed
Jun 19 17:00:27 dist-test-slave-3ffd krb5kdc[2325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888427, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:27.945000 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:27.921117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:38471 (local address 127.16.48.62:43495)
0619 17:00:27.921763 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:27.921772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:27.922943 (+ 1171us) server_negotiation.cc:407] Connection header received
0619 17:00:27.925925 (+ 2982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:27.925930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:27.926008 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:27.926125 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:27.928497 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:27.929590 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:27.932286 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:27.932759 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:27.932864 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:27.937103 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:27.937114 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:27.937121 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:27.937174 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:27.941003 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:27.941962 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:27.941971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:27.941978 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:27.942181 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:27.942690 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:27.942698 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:27.942704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:27.943143 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:27.943311 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:27.944086 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 17:00:27.944457 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":240,"threads_started":1}
I20260619 17:00:27.945721 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:27.921826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43495 (local address 127.16.48.2:38471)
0619 17:00:27.922756 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:27.922817 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:27.925530 (+ 2713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:27.926422 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:27.926439 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:27.927312 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:27.928231 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:27.928262 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:27.929793 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:27.929802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:27.932077 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:27.932098 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:27.933028 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:27.933038 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:27.933392 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:27.934191 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:00:27.934216 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:27.936734 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:27.941220 (+ 4486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:27.941230 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:27.941246 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:27.941772 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:27.942314 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:27.942318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:27.942320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:27.942544 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:27.943273 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:27.943279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:27.943657 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 17:00:27.943899 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 17:00:27.944461 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":349,"threads_started":1}
I20260619 17:00:27.947505 2693 heartbeater.cc:347] Connected to a master server at 127.16.48.62:43495
I20260619 17:00:27.947934 2693 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:27.949086 2693 heartbeater.cc:511] Master 127.16.48.62:43495 requested a full tablet report, sending...
I20260619 17:00:27.951136 2370 ts_manager.cc:194] Registered new tserver with Master: c3a45d45137e4b78999146bc9231285c (127.16.48.2:43965)
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
I20260619 17:00:27.952083 2370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:38471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 17:00:27.985805 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43495
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:28.342206 2699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:28.342590 2699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:28.342764 2699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:28.350659 2699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:28.350837 2699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:28.351028 2699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 17:00:28.361670 2699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:43495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:28.363086 2699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:28.364473 2699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:28.378903 2704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:28.379004 2705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:28.378978 2707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:28.379870 2699 server_base.cc:1034] running on GCE node
I20260619 17:00:28.380692 2699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:28.381816 2699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:28.383071 2699 hybrid_clock.cc:648] HybridClock initialized: now 1781888428383005 us; error 73 us; skew 500 ppm
I20260619 17:00:28.383474 2699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:28 dist-test-slave-3ffd krb5kdc[2325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888428, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:28.387192 2699 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 17:00:28.389030 2699 webserver.cc:460] Webserver started at http://127.16.48.3:43513/ using document root <none> and password file <none>
I20260619 17:00:28.389750 2699 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:28.389879 2699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:28.390185 2699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:28.394021 2699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "dc0ea97c2c664c47bd5a96034b5440ae"
format_stamp: "Formatted at 2026-06-19 17:00:28 on dist-test-slave-3ffd"
server_key: "b908551df234d341fabe11512ad8b006"
server_key_iv: "47f87ff81ab20213f29b58cadf7d8684"
server_key_version: "encryptionkey@0"
I20260619 17:00:28.395071 2699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "dc0ea97c2c664c47bd5a96034b5440ae"
format_stamp: "Formatted at 2026-06-19 17:00:28 on dist-test-slave-3ffd"
server_key: "b908551df234d341fabe11512ad8b006"
server_key_iv: "47f87ff81ab20213f29b58cadf7d8684"
server_key_version: "encryptionkey@0"
I20260619 17:00:28.401394 2699 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 17:00:28.405755 2714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:28.406697 2559 heartbeater.cc:503] Master 127.16.48.62:43495 was elected leader, sending a full tablet report...
I20260619 17:00:28.407630 2699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260619 17:00:28.407845 2699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "dc0ea97c2c664c47bd5a96034b5440ae"
format_stamp: "Formatted at 2026-06-19 17:00:28 on dist-test-slave-3ffd"
server_key: "b908551df234d341fabe11512ad8b006"
server_key_iv: "47f87ff81ab20213f29b58cadf7d8684"
server_key_version: "encryptionkey@0"
I20260619 17:00:28.408089 2699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:28.430260 2699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:28.435286 2699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:28.435632 2699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:28.436877 2699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:28.438421 2699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:28.438535 2699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:28.438702 2699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:28.438782 2699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:28.475503 2699 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:34923
I20260619 17:00:28.475546 2826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:34923 every 8 connection(s)
I20260619 17:00:28.476974 2699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 17:00:28.483090 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 2699
I20260619 17:00:28.483299 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 17:00:28.483876 16576 external_mini_cluster.cc:1428] Setting key 93227f37d81ef96bd0943b7b00f29a2c
Jun 19 17:00:28 dist-test-slave-3ffd krb5kdc[2325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888428, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:28.499819 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.479777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:40887 (local address 127.16.48.62:43495)
0619 17:00:28.480342 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.480348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.481374 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 17:00:28.483374 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.483379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.483486 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.483616 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:28.485897 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.486790 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.488586 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.489017 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.489158 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.493111 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:28.493117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:28.493120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:28.493160 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:28.495986 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:28.497013 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:28.497018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:28.497020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:28.497163 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:28.497778 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:28.497787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:28.497793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:28.498133 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:28.498348 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.499138 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.499394 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":244,"threads_started":1}
I20260619 17:00:28.500918 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.480372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:43495 (local address 127.16.48.3:40887)
0619 17:00:28.481188 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:28.481228 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:28.482989 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:28.483953 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:28.483967 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:28.484792 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:28.485685 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.485709 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.486978 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.486983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.488411 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.488427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.489285 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.489293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.489558 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.490337 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:00:28.490368 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:28.492815 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:28.496259 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:28.496269 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:28.496286 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:28.496824 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:28.497293 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:28.497297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:28.497300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:28.497606 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:28.498286 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:28.498294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:28.498706 (+ 412us) client_negotiation.cc:769] Sending connection context
0619 17:00:28.499026 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 17:00:28.499619 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":310,"threads_started":1}
I20260619 17:00:28.502475 2827 heartbeater.cc:347] Connected to a master server at 127.16.48.62:43495
I20260619 17:00:28.502821 2827 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:28.503604 2827 heartbeater.cc:511] Master 127.16.48.62:43495 requested a full tablet report, sending...
I20260619 17:00:28.505257 2370 ts_manager.cc:194] Registered new tserver with Master: dc0ea97c2c664c47bd5a96034b5440ae (127.16.48.3:34923)
I20260619 17:00:28.506147 2370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:40887
I20260619 17:00:28.512159 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 17:00:28.530903 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.516722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.16.48.62:43495)
0619 17:00:28.516906 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.516911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.517371 (+ 460us) server_negotiation.cc:407] Connection header received
0619 17:00:28.517521 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.517526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.517599 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.517700 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:28.519336 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.520233 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.521761 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.522169 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.522277 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.525258 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:28.525263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:28.525267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:28.525303 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:28.528198 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:28.529043 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:28.529047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:28.529050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:28.529184 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:28.529596 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:28.529599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:28.529607 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:28.529931 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:28.530147 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.530490 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.530669 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 17:00:28.535351 2370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 17:00:28.542912 2370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 17:00:28.572346 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.557842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:43965 (local address 127.0.0.1:38772)
0619 17:00:28.563501 (+ 5659us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:28.563529 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:28.563750 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:28.564769 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:28.564773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:28.564800 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:28.565173 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.565183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.566622 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.566628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.569569 (+ 2941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.569579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.571035 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.571042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.571264 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.571323 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 17:00:28.571516 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 17:00:28.571734 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5525,"mutex_wait_us":280,"thread_start_us":3895,"threads_started":1}
I20260619 17:00:28.573190 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.558181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.16.48.2:43965)
0619 17:00:28.562981 (+ 4800us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.562994 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.563635 (+ 641us) server_negotiation.cc:407] Connection header received
0619 17:00:28.563970 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.563979 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.564296 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.564480 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:28.565354 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.566449 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.569770 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.570793 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.570987 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.571227 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.572632 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.572857 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4567,"thread_start_us":4444,"threads_started":1}
I20260619 17:00:28.573470 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.556721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:34923 (local address 127.0.0.1:52508)
0619 17:00:28.557963 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:28.558027 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:28.558227 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:28.563354 (+ 5127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:28.563367 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:28.563408 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:28.564031 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.564053 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.566379 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.566386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.569551 (+ 3165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.569569 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.571482 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.571490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.572685 (+ 1195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.572732 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 17:00:28.572829 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 17:00:28.572986 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":110,"thread_start_us":6469,"threads_started":1}
I20260619 17:00:28.573534 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.557162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:39039 (local address 127.0.0.1:59230)
0619 17:00:28.562143 (+ 4981us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:28.562177 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:28.562483 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:28.563668 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:28.563673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:28.563692 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:28.564199 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.564212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.566338 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.566344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.569225 (+ 2881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.569240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.572302 (+ 3062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.572309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.572794 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.572832 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 17:00:28.572895 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 17:00:28.573005 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4798,"mutex_wait_us":179,"thread_start_us":1732,"threads_started":1}
I20260619 17:00:28.574208 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.558324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.16.48.1:39039)
0619 17:00:28.558997 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.559007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.562327 (+ 3320us) server_negotiation.cc:407] Connection header received
0619 17:00:28.562720 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.562727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.563111 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.563294 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:28.564406 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.566162 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.569428 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.571779 (+ 2351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.571924 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.572187 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.573194 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.573846 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":358,"threads_started":1}
I20260619 17:00:28.574433 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.556935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.16.48.3:34923)
0619 17:00:28.562572 (+ 5637us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.562581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.562617 (+ 36us) server_negotiation.cc:407] Connection header received
0619 17:00:28.562715 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.562724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.563037 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.564306 (+ 1269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:28.564630 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.566208 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.569743 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.571194 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.571403 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.571649 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.573862 (+ 2213us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.574058 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5375,"thread_start_us":5208,"threads_started":1}
I20260619 17:00:28.577458 2628 tablet_service.cc:1467] Processing CreateTablet for tablet fd0fc59ac08049aabc6cd3c2e258ce5a (DEFAULT_TABLE table=test-table [id=065dee660de841a7a18e1768cf9fe682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:28.578181 2494 tablet_service.cc:1467] Processing CreateTablet for tablet fd0fc59ac08049aabc6cd3c2e258ce5a (DEFAULT_TABLE table=test-table [id=065dee660de841a7a18e1768cf9fe682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:28.578274 2762 tablet_service.cc:1467] Processing CreateTablet for tablet fd0fc59ac08049aabc6cd3c2e258ce5a (DEFAULT_TABLE table=test-table [id=065dee660de841a7a18e1768cf9fe682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:28.579210 2628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0fc59ac08049aabc6cd3c2e258ce5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:28.579346 2494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0fc59ac08049aabc6cd3c2e258ce5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:28.579734 2762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0fc59ac08049aabc6cd3c2e258ce5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:28.592626 2846 tablet_bootstrap.cc:492] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: Bootstrap starting.
I20260619 17:00:28.596657 2846 tablet_bootstrap.cc:654] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:28.598009 2846 log.cc:824] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:28.601118 2848 tablet_bootstrap.cc:492] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: Bootstrap starting.
I20260619 17:00:28.601428 2846 tablet_bootstrap.cc:492] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: No bootstrap required, opened a new log
I20260619 17:00:28.601732 2846 ts_tablet_manager.cc:1397] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 17:00:28.604192 2847 tablet_bootstrap.cc:492] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: Bootstrap starting.
I20260619 17:00:28.604477 2848 tablet_bootstrap.cc:654] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:28.605968 2848 log.cc:824] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:28.609010 2847 tablet_bootstrap.cc:654] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:28.610658 2847 log.cc:824] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:28.611059 2848 tablet_bootstrap.cc:492] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: No bootstrap required, opened a new log
I20260619 17:00:28.611418 2848 ts_tablet_manager.cc:1397] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 17:00:28.612757 2846 raft_consensus.cc:348] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } }
I20260619 17:00:28.613281 2846 raft_consensus.cc:374] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:28.613392 2846 raft_consensus.cc:729] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de733238abed41c384dda03d2df56612, State: Initialized, Role: FOLLOWER
I20260619 17:00:28.614157 2846 consensus_queue.cc:260] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } }
I20260619 17:00:28.614800 2847 tablet_bootstrap.cc:492] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: No bootstrap required, opened a new log
I20260619 17:00:28.615156 2847 ts_tablet_manager.cc:1397] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260619 17:00:28.616320 2846 ts_tablet_manager.cc:1428] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260619 17:00:28.622634 2848 raft_consensus.cc:348] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } }
I20260619 17:00:28.623175 2848 raft_consensus.cc:374] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:28.623307 2848 raft_consensus.cc:729] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0ea97c2c664c47bd5a96034b5440ae, State: Initialized, Role: FOLLOWER
I20260619 17:00:28.623991 2848 consensus_queue.cc:260] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } }
W20260619 17:00:28.624570 2560 tablet.cc:2369] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:28.625043 2847 raft_consensus.cc:348] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } }
I20260619 17:00:28.625566 2847 raft_consensus.cc:374] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:28.625674 2827 heartbeater.cc:503] Master 127.16.48.62:43495 was elected leader, sending a full tablet report...
I20260619 17:00:28.625674 2847 raft_consensus.cc:729] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a45d45137e4b78999146bc9231285c, State: Initialized, Role: FOLLOWER
I20260619 17:00:28.626323 2847 consensus_queue.cc:260] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } }
I20260619 17:00:28.626518 2848 ts_tablet_manager.cc:1428] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260619 17:00:28.627525 2693 heartbeater.cc:503] Master 127.16.48.62:43495 was elected leader, sending a full tablet report...
I20260619 17:00:28.628360 2847 ts_tablet_manager.cc:1428] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260619 17:00:28.670110 2694 tablet.cc:2369] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:28.686443 2854 raft_consensus.cc:482] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 17:00:28.686852 2854 raft_consensus.cc:504] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } }
I20260619 17:00:28.688941 2854 leader_election.cc:288] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc0ea97c2c664c47bd5a96034b5440ae (127.16.48.3:34923), de733238abed41c384dda03d2df56612 (127.16.48.1:39039)
I20260619 17:00:28.697870 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.689458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:60579 (local address 127.16.48.1:39039)
0619 17:00:28.689747 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.689756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.690163 (+ 407us) server_negotiation.cc:407] Connection header received
0619 17:00:28.690494 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.690499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.690570 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.690687 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:28.691473 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.692552 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.694930 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.696403 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.696580 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.696720 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.697346 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.697536 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 17:00:28.697949 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.689460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:39247 (local address 127.16.48.3:34923)
0619 17:00:28.689731 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.689737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.690612 (+ 875us) server_negotiation.cc:407] Connection header received
0619 17:00:28.690820 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.690824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.690885 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.690991 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:28.691744 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.692787 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.694887 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.696146 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.696328 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.696477 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.697288 (+ 811us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.697510 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 17:00:28.697930 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.689275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:34923 (local address 127.16.48.2:39247)
0619 17:00:28.690492 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:28.690508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:28.690678 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:28.691009 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:28.691013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:28.691062 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:28.691526 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.691539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.692940 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.692945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.694670 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.694692 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.696544 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.696554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.696941 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.697038 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 17:00:28.697193 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 17:00:28.697421 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":557,"thread_start_us":397,"threads_started":1}
I20260619 17:00:28.699781 2782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0fc59ac08049aabc6cd3c2e258ce5a" candidate_uuid: "c3a45d45137e4b78999146bc9231285c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" is_pre_election: true
I20260619 17:00:28.700554 2782 raft_consensus.cc:2454] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3a45d45137e4b78999146bc9231285c in term 0.
I20260619 17:00:28.701160 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.689275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:39039 (local address 127.16.48.2:60579)
0619 17:00:28.690018 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:28.690039 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:28.690314 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:28.690707 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:28.690713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:28.690742 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:28.691242 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.691258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.692744 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.692750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.694699 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:28.694716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.696702 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.696712 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:28.696990 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.697066 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 17:00:28.697186 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 17:00:28.698478 (+ 1292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":1092,"thread_start_us":362,"threads_started":1}
I20260619 17:00:28.701637 2583 leader_election.cc:302] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3a45d45137e4b78999146bc9231285c, dc0ea97c2c664c47bd5a96034b5440ae; no voters:
I20260619 17:00:28.702355 2854 raft_consensus.cc:2781] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 17:00:28.702539 2854 raft_consensus.cc:482] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 17:00:28.702656 2854 raft_consensus.cc:3037] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:28.702553 2514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0fc59ac08049aabc6cd3c2e258ce5a" candidate_uuid: "c3a45d45137e4b78999146bc9231285c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de733238abed41c384dda03d2df56612" is_pre_election: true
I20260619 17:00:28.703235 2514 raft_consensus.cc:2454] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3a45d45137e4b78999146bc9231285c in term 0.
I20260619 17:00:28.705462 2854 raft_consensus.cc:504] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } }
I20260619 17:00:28.706297 2854 leader_election.cc:288] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [CANDIDATE]: Term 1 election: Requested vote from peers dc0ea97c2c664c47bd5a96034b5440ae (127.16.48.3:34923), de733238abed41c384dda03d2df56612 (127.16.48.1:39039)
I20260619 17:00:28.706967 2782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0fc59ac08049aabc6cd3c2e258ce5a" candidate_uuid: "c3a45d45137e4b78999146bc9231285c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0ea97c2c664c47bd5a96034b5440ae"
I20260619 17:00:28.707007 2514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0fc59ac08049aabc6cd3c2e258ce5a" candidate_uuid: "c3a45d45137e4b78999146bc9231285c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de733238abed41c384dda03d2df56612"
I20260619 17:00:28.707242 2514 raft_consensus.cc:3037] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:28.707234 2782 raft_consensus.cc:3037] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:28.709692 2514 raft_consensus.cc:2454] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3a45d45137e4b78999146bc9231285c in term 1.
I20260619 17:00:28.709939 2782 raft_consensus.cc:2454] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3a45d45137e4b78999146bc9231285c in term 1.
I20260619 17:00:28.710341 2584 leader_election.cc:302] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3a45d45137e4b78999146bc9231285c, de733238abed41c384dda03d2df56612; no voters:
I20260619 17:00:28.710748 2854 raft_consensus.cc:2781] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:28.711470 2854 raft_consensus.cc:686] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 1 LEADER]: Becoming Leader. State: Replica: c3a45d45137e4b78999146bc9231285c, State: Running, Role: LEADER
I20260619 17:00:28.712033 2854 consensus_queue.cc:237] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } }
I20260619 17:00:28.717242 2369 catalog_manager.cc:5495] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c reported cstate change: term changed from 0 to 1, leader changed from <none> to c3a45d45137e4b78999146bc9231285c (127.16.48.2). New cstate: current_term: 1 leader_uuid: "c3a45d45137e4b78999146bc9231285c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3a45d45137e4b78999146bc9231285c" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 43965 } health_report { overall_health: HEALTHY } } }
W20260619 17:00:28.728637 2828 tablet.cc:2369] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:28.737112 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.730985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.16.48.2:43965)
0619 17:00:28.731237 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.731243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.731265 (+ 22us) server_negotiation.cc:407] Connection header received
0619 17:00:28.731437 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.731441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.731522 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.731687 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:00:28.732186 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.732915 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.734345 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.734762 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.734907 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.735608 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:00:28.736282 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:00:28.736486 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.736716 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.736780 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 17:00:28.746372 2782 raft_consensus.cc:1261] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [term 1 FOLLOWER]: Refusing update from remote peer c3a45d45137e4b78999146bc9231285c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:28.746369 2514 raft_consensus.cc:1261] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [term 1 FOLLOWER]: Refusing update from remote peer c3a45d45137e4b78999146bc9231285c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:28.747632 2854 consensus_queue.cc:1035] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de733238abed41c384dda03d2df56612" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 39039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:28.748027 2857 consensus_queue.cc:1035] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0ea97c2c664c47bd5a96034b5440ae" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:28.761562 2860 mvcc.cc:205] Tried to move back new op lower bound from 7298615004132855808 to 7298615004009189376. Current Snapshot: MvccSnapshot[applied={T|T < 7298615004132855808}]
I20260619 17:00:28.762135 2862 mvcc.cc:205] Tried to move back new op lower bound from 7298615004132855808 to 7298615004009189376. Current Snapshot: MvccSnapshot[applied={T|T < 7298615004132855808}]
I20260619 17:00:28.776747 2370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:28.777088 2370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:28.781380 2370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91c0d922a9f04af9bdca8a7b2ca818bc: Sending DeleteTablet for 3 replicas of tablet fd0fc59ac08049aabc6cd3c2e258ce5a
I20260619 17:00:28.782819 2494 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0fc59ac08049aabc6cd3c2e258ce5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:28 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:59230
I20260619 17:00:28.782860 2762 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0fc59ac08049aabc6cd3c2e258ce5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:28 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:52508
I20260619 17:00:28.782955 2628 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0fc59ac08049aabc6cd3c2e258ce5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:28 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:38772
I20260619 17:00:28.785310 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 2431
I20260619 17:00:28.801618 2869 tablet_replica.cc:330] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: stopping tablet replica
I20260619 17:00:28.802342 2869 raft_consensus.cc:2229] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:28.803237 2869 raft_consensus.cc:2258] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:28.804401 2871 tablet_replica.cc:330] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: stopping tablet replica
I20260619 17:00:28.804987 2871 raft_consensus.cc:2229] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 1 LEADER]: Raft consensus shutting down.
I20260619 17:00:28.805769 2871 raft_consensus.cc:2258] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:28.807165 2869 ts_tablet_manager.cc:1905] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:28.808375 2871 ts_tablet_manager.cc:1905] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 17:00:28 dist-test-slave-3ffd krb5kdc[2325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 17:00:28.812968 2870 tablet_replica.cc:330] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: stopping tablet replica
I20260619 17:00:28.813520 2870 raft_consensus.cc:2229] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:28.814235 2870 raft_consensus.cc:2258] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:28.815935 2869 ts_tablet_manager.cc:1918] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:28.816313 2869 log.cc:1196] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/fd0fc59ac08049aabc6cd3c2e258ce5a
I20260619 17:00:28.818056 2870 ts_tablet_manager.cc:1905] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:28.818204 2871 ts_tablet_manager.cc:1918] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:28.818410 2871 log.cc:1196] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/fd0fc59ac08049aabc6cd3c2e258ce5a
I20260619 17:00:28.818989 2869 ts_tablet_manager.cc:1939] T fd0fc59ac08049aabc6cd3c2e258ce5a P de733238abed41c384dda03d2df56612: Deleting consensus metadata
I20260619 17:00:28.819082 2871 ts_tablet_manager.cc:1939] T fd0fc59ac08049aabc6cd3c2e258ce5a P c3a45d45137e4b78999146bc9231285c: Deleting consensus metadata
I20260619 17:00:28.821409 2355 catalog_manager.cc:4841] TS c3a45d45137e4b78999146bc9231285c (127.16.48.2:43965): tablet fd0fc59ac08049aabc6cd3c2e258ce5a (table test-table [id=065dee660de841a7a18e1768cf9fe682]) successfully deleted
I20260619 17:00:28.823006 2357 catalog_manager.cc:4841] TS de733238abed41c384dda03d2df56612 (127.16.48.1:39039): tablet fd0fc59ac08049aabc6cd3c2e258ce5a (table test-table [id=065dee660de841a7a18e1768cf9fe682]) successfully deleted
I20260619 17:00:28.824755 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.788090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.16.48.1:39039)
0619 17:00:28.788306 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.788313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.791568 (+ 3255us) server_negotiation.cc:407] Connection header received
0619 17:00:28.792662 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.792672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.792793 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.792892 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:28.798435 (+ 5543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.800146 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.804852 (+ 4706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.805615 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.805757 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.816185 (+ 10428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:28.816208 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:28.816229 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:28.816283 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:28.820760 (+ 4477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:28.822097 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:28.822106 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:28.822108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:28.822278 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:28.822760 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:28.822765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:28.822768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:28.823262 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:28.823577 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.824088 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.824365 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 17:00:28.825603 2554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:28.829440 2870 ts_tablet_manager.cc:1918] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:28.829648 2870 log.cc:1196] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/fd0fc59ac08049aabc6cd3c2e258ce5a
I20260619 17:00:28.830191 2870 ts_tablet_manager.cc:1939] T fd0fc59ac08049aabc6cd3c2e258ce5a P dc0ea97c2c664c47bd5a96034b5440ae: Deleting consensus metadata
I20260619 17:00:28.832333 2356 catalog_manager.cc:4841] TS dc0ea97c2c664c47bd5a96034b5440ae (127.16.48.3:34923): tablet fd0fc59ac08049aabc6cd3c2e258ce5a (table test-table [id=065dee660de841a7a18e1768cf9fe682]) successfully deleted
I20260619 17:00:28.920744 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 2431
I20260619 17:00:28.945870 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 2565
Jun 19 17:00:28 dist-test-slave-3ffd krb5kdc[2325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 17:00:28.966215 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:28.946740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.16.48.2:43965)
0619 17:00:28.946925 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:28.946931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:28.947004 (+ 73us) server_negotiation.cc:407] Connection header received
0619 17:00:28.947225 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:28.947230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:28.947310 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:28.947430 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:28.949172 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.950506 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.952580 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:28.953324 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:28.953472 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:28.959116 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:28.959128 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:28.959144 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:28.959180 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:28.962655 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:28.963587 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:28.963595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:28.963597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:28.963768 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:28.964432 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:28.964436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:28.964438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:28.964814 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:28.965237 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:28.965578 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 17:00:28.965839 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 17:00:28.966984 2688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:29.086261 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 2565
I20260619 17:00:29.102986 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 2699
Jun 19 17:00:29 dist-test-slave-3ffd krb5kdc[2325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 17:00:29.119436 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:29.103815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.16.48.3:34923)
0619 17:00:29.104003 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:29.104008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:29.104027 (+ 19us) server_negotiation.cc:407] Connection header received
0619 17:00:29.104181 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:29.104185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:29.104263 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:29.104349 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:29.105757 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:29.106492 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:29.107854 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:29.108345 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:29.108449 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:29.113230 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:29.113248 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:29.113264 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:29.113308 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:29.116738 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:29.117583 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:29.117592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:29.117594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:29.117766 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:29.118122 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:29.118125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:29.118127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:29.118483 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:29.118715 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:29.118976 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 17:00:29.119168 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 17:00:29.120193 2822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:29.212255 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 2699
I20260619 17:00:29.229753 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 2338
I20260619 17:00:29.230834 2400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:29.345180 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 2338
2026-06-19T17:00:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4560 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T17:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:29Z Disabled control of system clock
I20260619 17:00:29.425874 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:44267
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:44267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 17:00:29.780604 2880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:29.781025 2880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:29.781140 2880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:29.788628 2880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:29.788746 2880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:29.788785 2880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:29.788843 2880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:29.788889 2880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:29.798540 2880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:44267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:44267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:29.799855 2880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:29.801796 2880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:29.814702 2885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:29.814656 2886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:29.814738 2888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:29.815764 2880 server_base.cc:1034] running on GCE node
I20260619 17:00:29.816754 2880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:29.818378 2880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:29.819621 2880 hybrid_clock.cc:648] HybridClock initialized: now 1781888429819550 us; error 62 us; skew 500 ppm
I20260619 17:00:29.820030 2880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:29.822597 2880 webserver.cc:460] Webserver started at http://127.16.48.62:39311/ using document root <none> and password file <none>
I20260619 17:00:29.823375 2880 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:29.823499 2880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:29.823860 2880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:29.827116 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "a6dfd4258e5142f5aa26be7941db4d80"
format_stamp: "Formatted at 2026-06-19 17:00:29 on dist-test-slave-3ffd"
server_key: "2fcd147e20f173c312eb9447b6aa22a4"
server_key_iv: "cb7345e8e010b3bb35b1284fcc2dac8c"
server_key_version: "encryptionkey@0"
I20260619 17:00:29.828073 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "a6dfd4258e5142f5aa26be7941db4d80"
format_stamp: "Formatted at 2026-06-19 17:00:29 on dist-test-slave-3ffd"
server_key: "2fcd147e20f173c312eb9447b6aa22a4"
server_key_iv: "cb7345e8e010b3bb35b1284fcc2dac8c"
server_key_version: "encryptionkey@0"
I20260619 17:00:29.833917 2880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 17:00:29.837826 2894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:29.839620 2880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 17:00:29.839860 2880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "a6dfd4258e5142f5aa26be7941db4d80"
format_stamp: "Formatted at 2026-06-19 17:00:29 on dist-test-slave-3ffd"
server_key: "2fcd147e20f173c312eb9447b6aa22a4"
server_key_iv: "cb7345e8e010b3bb35b1284fcc2dac8c"
server_key_version: "encryptionkey@0"
I20260619 17:00:29.840126 2880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:29.854276 2880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:29.855237 2880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:29.855579 2880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:29.876101 2880 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:44267
I20260619 17:00:29.876103 2945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:44267 every 8 connection(s)
I20260619 17:00:29.877671 2880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:29.882051 2946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:29.883257 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 2880
I20260619 17:00:29.883450 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:29.883937 16576 external_mini_cluster.cc:1428] Setting key 05e73e540adb59e938c1be6d9c80088e
I20260619 17:00:29.898494 2946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80: Bootstrap starting.
I20260619 17:00:29.900946 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:29.886256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.16.48.62:44267)
0619 17:00:29.887573 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:29.887590 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:29.887686 (+ 96us) server_negotiation.cc:407] Connection header received
0619 17:00:29.889607 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:29.889642 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:29.890428 (+ 786us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:29.890950 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:29.891544 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:29.893521 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:29.895057 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:29.896296 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:29.896510 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:29.897746 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:29.897756 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:29.897780 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:29.898574 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:29.898692 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:29.898719 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:29.898834 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:29.899067 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 17:00:29.899496 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":391,"threads_started":1}
I20260619 17:00:29.902587 2946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:29.903788 2946 log.cc:824] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:29.906949 2946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80: No bootstrap required, opened a new log
I20260619 17:00:29.914752 2946 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6dfd4258e5142f5aa26be7941db4d80" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44267 } }
I20260619 17:00:29.915284 2946 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:29.915509 2946 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6dfd4258e5142f5aa26be7941db4d80, State: Initialized, Role: FOLLOWER
I20260619 17:00:29.916342 2946 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6dfd4258e5142f5aa26be7941db4d80" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44267 } }
I20260619 17:00:29.916765 2946 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:29.916929 2946 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:29.917150 2946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:29.919641 2946 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6dfd4258e5142f5aa26be7941db4d80" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44267 } }
I20260619 17:00:29.920415 2946 leader_election.cc:302] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6dfd4258e5142f5aa26be7941db4d80; no voters:
I20260619 17:00:29.921144 2946 leader_election.cc:288] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:29.921317 2951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:29.922092 2951 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [term 1 LEADER]: Becoming Leader. State: Replica: a6dfd4258e5142f5aa26be7941db4d80, State: Running, Role: LEADER
I20260619 17:00:29.922726 2951 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6dfd4258e5142f5aa26be7941db4d80" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44267 } }
I20260619 17:00:29.923723 2946 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:29.926020 2952 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6dfd4258e5142f5aa26be7941db4d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6dfd4258e5142f5aa26be7941db4d80" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44267 } } }
I20260619 17:00:29.926570 2952 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:29.927667 2958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:29.928985 2953 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6dfd4258e5142f5aa26be7941db4d80. Latest consensus state: current_term: 1 leader_uuid: "a6dfd4258e5142f5aa26be7941db4d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6dfd4258e5142f5aa26be7941db4d80" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44267 } } }
I20260619 17:00:29.929365 2953 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:29.933743 2958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:29.945624 2958 catalog_manager.cc:1347] Generated new cluster ID: cf3f4698008a42e884d92116a1b2157b
I20260619 17:00:29.945834 2958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:29.961341 2958 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:29.963291 2958 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:29.970717 2958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80: Generated new TSK 0
I20260619 17:00:29.971700 2958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:29.978133 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44267
--builtin_ntp_servers=127.16.48.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 17:00:30.369880 2970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:30.370271 2970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:30.370437 2970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:30.378217 2970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:30.378407 2970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:30.378633 2970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 17:00:30.388657 2970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:30.390105 2970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:30.392023 2970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:30.406545 2976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:30.406694 2978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:30.406695 2975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:30.413396 2970 server_base.cc:1034] running on GCE node
I20260619 17:00:30.414206 2970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:30.415390 2970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:30.416682 2970 hybrid_clock.cc:648] HybridClock initialized: now 1781888430416624 us; error 57 us; skew 500 ppm
I20260619 17:00:30.417124 2970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:30.425637 2970 webserver.cc:460] Webserver started at http://127.16.48.1:40901/ using document root <none> and password file <none>
I20260619 17:00:30.426482 2970 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:30.426674 2970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:30.427028 2970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:30.430342 2970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "538d3e696ae54e31b5e134758513418e"
format_stamp: "Formatted at 2026-06-19 17:00:30 on dist-test-slave-3ffd"
server_key: "d700179f5dbba2d5cd65d03ba9326a28"
server_key_iv: "c50d22dfdb93fe782995ec58c8484c68"
server_key_version: "encryptionkey@0"
I20260619 17:00:30.431296 2970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "538d3e696ae54e31b5e134758513418e"
format_stamp: "Formatted at 2026-06-19 17:00:30 on dist-test-slave-3ffd"
server_key: "d700179f5dbba2d5cd65d03ba9326a28"
server_key_iv: "c50d22dfdb93fe782995ec58c8484c68"
server_key_version: "encryptionkey@0"
I20260619 17:00:30.437726 2970 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 17:00:30.442070 2984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:30.443940 2970 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 17:00:30.444233 2970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "538d3e696ae54e31b5e134758513418e"
format_stamp: "Formatted at 2026-06-19 17:00:30 on dist-test-slave-3ffd"
server_key: "d700179f5dbba2d5cd65d03ba9326a28"
server_key_iv: "c50d22dfdb93fe782995ec58c8484c68"
server_key_version: "encryptionkey@0"
I20260619 17:00:30.444506 2970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:30.465421 2970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:30.466341 2970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:30.466660 2970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:30.467765 2970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:30.469457 2970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:30.469599 2970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:30.469729 2970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:30.469816 2970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:30.505525 2970 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:45395
I20260619 17:00:30.505556 3096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:45395 every 8 connection(s)
I20260619 17:00:30.506976 2970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 17:00:30.515594 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 2970
I20260619 17:00:30.515836 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 17:00:30.516451 16576 external_mini_cluster.cc:1428] Setting key fd2a3db5779188ffe74ffa1183184002
I20260619 17:00:30.519618 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44267
--builtin_ntp_servers=127.16.48.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 17:00:30.520918 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:30.509734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:50925 (local address 127.16.48.62:44267)
0619 17:00:30.510217 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:30.510221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:30.511382 (+ 1161us) server_negotiation.cc:407] Connection header received
0619 17:00:30.513355 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:30.513361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:30.513475 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:30.513614 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:30.514773 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:30.515512 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:30.517501 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:30.517946 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:30.518093 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:30.519662 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:30.519667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:30.519670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:30.519691 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:30.519731 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:30.519743 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:30.519850 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:30.520221 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 17:00:30.520368 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":266,"threads_started":1}
I20260619 17:00:30.521751 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:30.510228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44267 (local address 127.16.48.1:50925)
0619 17:00:30.511203 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:30.511247 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:30.513056 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:30.513868 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:30.513879 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:30.513977 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:30.514591 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:30.514610 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:30.515693 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:30.515699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:30.517321 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:30.517336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:30.518202 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:30.518209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:30.518482 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:30.519265 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 17:00:30.519268 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:30.519315 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 17:00:30.519421 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:30.519910 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:30.519923 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:30.519935 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 17:00:30.520167 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 17:00:30.520554 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":335,"threads_started":1}
I20260619 17:00:30.523695 3097 heartbeater.cc:347] Connected to a master server at 127.16.48.62:44267
I20260619 17:00:30.524081 3097 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:30.524986 3097 heartbeater.cc:511] Master 127.16.48.62:44267 requested a full tablet report, sending...
I20260619 17:00:30.527709 2911 ts_manager.cc:194] Registered new tserver with Master: 538d3e696ae54e31b5e134758513418e (127.16.48.1:45395)
I20260619 17:00:30.529493 2911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.1:50925
W20260619 17:00:30.880584 3101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:30.880935 3101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:30.881115 3101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:30.889048 3101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:30.889215 3101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:30.889405 3101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 17:00:30.899483 3101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:30.900959 3101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:30.902902 3101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:30.917239 3109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:30.917290 3106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:30.917680 3107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:30.918292 3101 server_base.cc:1034] running on GCE node
I20260619 17:00:30.919008 3101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:30.920097 3101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:30.921394 3101 hybrid_clock.cc:648] HybridClock initialized: now 1781888430921331 us; error 60 us; skew 500 ppm
I20260619 17:00:30.921700 3101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:30.924216 3101 webserver.cc:460] Webserver started at http://127.16.48.2:37475/ using document root <none> and password file <none>
I20260619 17:00:30.925107 3101 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:30.925246 3101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:30.925588 3101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:30.928876 3101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "8900d129b99a4d25ba7a5ad10adff670"
format_stamp: "Formatted at 2026-06-19 17:00:30 on dist-test-slave-3ffd"
server_key: "4b92d24d4257737b044ef77260e212f0"
server_key_iv: "25502b5f17bffc05c43e272785ba1f57"
server_key_version: "encryptionkey@0"
I20260619 17:00:30.929831 3101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "8900d129b99a4d25ba7a5ad10adff670"
format_stamp: "Formatted at 2026-06-19 17:00:30 on dist-test-slave-3ffd"
server_key: "4b92d24d4257737b044ef77260e212f0"
server_key_iv: "25502b5f17bffc05c43e272785ba1f57"
server_key_version: "encryptionkey@0"
I20260619 17:00:30.935936 3101 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 17:00:30.940286 3115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:30.941985 3101 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 17:00:30.942206 3101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "8900d129b99a4d25ba7a5ad10adff670"
format_stamp: "Formatted at 2026-06-19 17:00:30 on dist-test-slave-3ffd"
server_key: "4b92d24d4257737b044ef77260e212f0"
server_key_iv: "25502b5f17bffc05c43e272785ba1f57"
server_key_version: "encryptionkey@0"
I20260619 17:00:30.942462 3101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:30.971347 3101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:30.972429 3101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:30.972725 3101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:30.973783 3101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:30.975354 3101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:30.975488 3101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:30.975630 3101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:30.975720 3101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:31.010495 3101 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:44099
I20260619 17:00:31.010596 3234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:44099 every 8 connection(s)
I20260619 17:00:31.012015 3101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 17:00:31.020416 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 3101
I20260619 17:00:31.020624 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 17:00:31.021214 16576 external_mini_cluster.cc:1428] Setting key 61b8f867687d59512e64dd584ac838da
I20260619 17:00:31.024483 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44267
--builtin_ntp_servers=127.16.48.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 17:00:31.027638 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.014799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:35569 (local address 127.16.48.62:44267)
0619 17:00:31.015006 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:31.015011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:31.016622 (+ 1611us) server_negotiation.cc:407] Connection header received
0619 17:00:31.018655 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:31.018663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:31.018788 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:31.018917 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:31.020578 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.021487 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.023955 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.024779 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.024946 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.026594 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:31.026603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:31.026611 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:31.026639 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:31.026687 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:31.026707 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:31.026811 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:31.027057 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 17:00:31.027187 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 17:00:31.028680 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.015332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44267 (local address 127.16.48.2:35569)
0619 17:00:31.016441 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:31.016492 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:31.018242 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:31.019261 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:31.019281 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:31.019436 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:31.020293 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.020331 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.021770 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.021776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.023760 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.023776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.025067 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.025074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.025330 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.026151 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 17:00:31.026154 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:31.026205 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 17:00:31.026331 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:31.026818 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:31.026827 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:31.026833 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 17:00:31.027006 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 17:00:31.027366 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":314,"threads_started":1}
I20260619 17:00:31.030305 3235 heartbeater.cc:347] Connected to a master server at 127.16.48.62:44267
I20260619 17:00:31.030694 3235 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:31.031718 3235 heartbeater.cc:511] Master 127.16.48.62:44267 requested a full tablet report, sending...
I20260619 17:00:31.033663 2911 ts_manager.cc:194] Registered new tserver with Master: 8900d129b99a4d25ba7a5ad10adff670 (127.16.48.2:44099)
I20260619 17:00:31.034996 2911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.2:35569
W20260619 17:00:31.385354 3238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:31.385691 3238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:31.385789 3238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:31.393734 3238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:31.393869 3238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:31.393983 3238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 17:00:31.403791 3238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:31.405256 3238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:31.407058 3238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:31.421633 3244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:31.421710 3246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:31.421729 3243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:31.423345 3238 server_base.cc:1034] running on GCE node
I20260619 17:00:31.424216 3238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:31.425520 3238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:31.426777 3238 hybrid_clock.cc:648] HybridClock initialized: now 1781888431426715 us; error 55 us; skew 500 ppm
I20260619 17:00:31.427165 3238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:31.429764 3238 webserver.cc:460] Webserver started at http://127.16.48.3:35101/ using document root <none> and password file <none>
I20260619 17:00:31.430565 3238 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:31.430691 3238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:31.431046 3238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:31.434432 3238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "ed3fca694e3c4a48944dde6d1d415b88"
format_stamp: "Formatted at 2026-06-19 17:00:31 on dist-test-slave-3ffd"
server_key: "5b8445d4c04bf1b960844981528aa934"
server_key_iv: "28853a53ee71ec2537732bd054810dbf"
server_key_version: "encryptionkey@0"
I20260619 17:00:31.435384 3238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "ed3fca694e3c4a48944dde6d1d415b88"
format_stamp: "Formatted at 2026-06-19 17:00:31 on dist-test-slave-3ffd"
server_key: "5b8445d4c04bf1b960844981528aa934"
server_key_iv: "28853a53ee71ec2537732bd054810dbf"
server_key_version: "encryptionkey@0"
I20260619 17:00:31.441365 3238 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 17:00:31.445683 3252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:31.447487 3238 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 17:00:31.447710 3238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "ed3fca694e3c4a48944dde6d1d415b88"
format_stamp: "Formatted at 2026-06-19 17:00:31 on dist-test-slave-3ffd"
server_key: "5b8445d4c04bf1b960844981528aa934"
server_key_iv: "28853a53ee71ec2537732bd054810dbf"
server_key_version: "encryptionkey@0"
I20260619 17:00:31.447942 3238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:31.462209 3238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:31.463152 3238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:31.463469 3238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:31.464607 3238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:31.466223 3238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:31.466315 3238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:31.466455 3238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:31.466539 3238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:31.502096 3238 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:40915
I20260619 17:00:31.502156 3364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:40915 every 8 connection(s)
I20260619 17:00:31.503618 3238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 17:00:31.505676 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 3238
I20260619 17:00:31.505865 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 17:00:31.506467 16576 external_mini_cluster.cc:1428] Setting key 71ae6ffeea61db934aae63ab78a0831e
I20260619 17:00:31.518020 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.506918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:48703 (local address 127.16.48.62:44267)
0619 17:00:31.507136 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:31.507143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:31.508993 (+ 1850us) server_negotiation.cc:407] Connection header received
0619 17:00:31.511569 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:31.511573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:31.511645 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:31.511730 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:31.512927 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.513681 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.515269 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.515678 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.515779 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.517303 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:31.517308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:31.517311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:31.517333 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:31.517365 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:31.517377 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:31.517450 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:31.517666 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 17:00:31.517756 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 17:00:31.519189 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.507527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44267 (local address 127.16.48.3:48703)
0619 17:00:31.508822 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:31.508872 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:31.511250 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:31.511976 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:31.511984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:31.512104 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:31.512765 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.512783 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.513832 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.513836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.515134 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.515148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.515873 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.515877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.516177 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.516964 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 17:00:31.516967 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:31.517012 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 17:00:31.517110 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:31.517463 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:31.517470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:31.517475 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 17:00:31.517626 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 17:00:31.517960 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":458,"threads_started":1}
I20260619 17:00:31.520959 3365 heartbeater.cc:347] Connected to a master server at 127.16.48.62:44267
I20260619 17:00:31.521327 3365 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:31.522197 3365 heartbeater.cc:511] Master 127.16.48.62:44267 requested a full tablet report, sending...
I20260619 17:00:31.523711 2911 ts_manager.cc:194] Registered new tserver with Master: ed3fca694e3c4a48944dde6d1d415b88 (127.16.48.3:40915)
I20260619 17:00:31.524709 2911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.3:48703
I20260619 17:00:31.533826 3097 heartbeater.cc:503] Master 127.16.48.62:44267 was elected leader, sending a full tablet report...
I20260619 17:00:31.535324 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 17:00:31.546332 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.539758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.16.48.62:44267)
0619 17:00:31.540026 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:31.540034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:31.540392 (+ 358us) server_negotiation.cc:407] Connection header received
0619 17:00:31.540613 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:31.540623 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:31.540702 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:31.540819 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:31.541335 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.542046 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.543549 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.544018 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.544235 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.545510 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:31.545515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:31.545519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:31.545540 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:31.545579 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:31.545592 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:31.545749 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:31.545952 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 17:00:31.546016 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 17:00:31.551209 2910 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 17:00:31.558353 2910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 17:00:31.597962 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.571911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45395 (local address 127.0.0.1:56090)
0619 17:00:31.587453 (+ 15542us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:31.587476 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:31.587651 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:31.590040 (+ 2389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:31.590046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:31.590075 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:31.590417 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.590428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.592210 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.592220 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.594381 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.594396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.596763 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.596768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.597012 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.597205 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 17:00:31.597345 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 17:00:31.597565 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15354,"mutex_wait_us":103,"thread_start_us":14055,"threads_started":1}
I20260619 17:00:31.598016 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.571989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.16.48.2:44099)
0619 17:00:31.587295 (+ 15306us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:31.587307 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:31.587348 (+ 41us) server_negotiation.cc:407] Connection header received
0619 17:00:31.587472 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:31.587482 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:31.587799 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:31.589355 (+ 1556us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:31.589562 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.591178 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.594447 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.596160 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.596303 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.596579 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:31.597494 (+ 915us) server_negotiation.cc:299] Negotiation successful
0619 17:00:31.597667 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15086,"thread_start_us":16014,"threads_started":1}
I20260619 17:00:31.598461 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.575327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.16.48.3:40915)
0619 17:00:31.587301 (+ 11974us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:31.587313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:31.588074 (+ 761us) server_negotiation.cc:407] Connection header received
0619 17:00:31.588261 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:31.588268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:31.588507 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:31.588718 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:31.589788 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.591373 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.596086 (+ 4713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.597161 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.597325 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.597616 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:31.597748 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 17:00:31.597914 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11785,"thread_start_us":11618,"threads_started":1}
I20260619 17:00:31.598861 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.572292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.16.48.1:45395)
0619 17:00:31.589467 (+ 17175us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:31.589477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:31.589520 (+ 43us) server_negotiation.cc:407] Connection header received
0619 17:00:31.589631 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:31.589640 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:31.589847 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:31.590064 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:31.590589 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.592042 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.594588 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.596157 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.596303 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.596648 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:31.598299 (+ 1651us) server_negotiation.cc:299] Negotiation successful
0619 17:00:31.598476 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16961,"thread_start_us":16805,"threads_started":1}
I20260619 17:00:31.598349 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.574511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:40915 (local address 127.0.0.1:45146)
0619 17:00:31.587776 (+ 13265us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:31.587795 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:31.587931 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:31.589015 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:31.589022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:31.589054 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:31.589566 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.589579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.591543 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.591547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.595816 (+ 4269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.595830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.597389 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.597395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.597557 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.597605 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 17:00:31.597706 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 17:00:31.597847 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13130,"mutex_wait_us":2363,"thread_start_us":14242,"threads_started":1}
I20260619 17:00:31.598829 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.571745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:44099 (local address 127.0.0.1:49634)
0619 17:00:31.585334 (+ 13589us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:31.585404 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:31.585705 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:31.588434 (+ 2729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:31.588447 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:31.588494 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:31.589053 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.589086 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.591344 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.591351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.594191 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.594210 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.596724 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.596733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.597027 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.597204 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 17:00:31.597345 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 17:00:31.598482 (+ 1137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13148,"mutex_wait_us":175,"thread_start_us":15440,"threads_started":1}
I20260619 17:00:31.603042 3300 tablet_service.cc:1467] Processing CreateTablet for tablet ae7d9078c1d0469a9c43facd7b5f8190 (DEFAULT_TABLE table=test-table [id=56f9c9a1abd64c09b286d4c8d7dcf644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:31.603279 3170 tablet_service.cc:1467] Processing CreateTablet for tablet ae7d9078c1d0469a9c43facd7b5f8190 (DEFAULT_TABLE table=test-table [id=56f9c9a1abd64c09b286d4c8d7dcf644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:31.603765 3032 tablet_service.cc:1467] Processing CreateTablet for tablet ae7d9078c1d0469a9c43facd7b5f8190 (DEFAULT_TABLE table=test-table [id=56f9c9a1abd64c09b286d4c8d7dcf644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:31.604635 3300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7d9078c1d0469a9c43facd7b5f8190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:31.604667 3170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7d9078c1d0469a9c43facd7b5f8190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:31.605219 3032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7d9078c1d0469a9c43facd7b5f8190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:31.617913 3383 tablet_bootstrap.cc:492] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: Bootstrap starting.
I20260619 17:00:31.620949 3384 tablet_bootstrap.cc:492] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: Bootstrap starting.
I20260619 17:00:31.622159 3383 tablet_bootstrap.cc:654] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:31.623155 3385 tablet_bootstrap.cc:492] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: Bootstrap starting.
I20260619 17:00:31.624482 3383 log.cc:824] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:31.625284 3384 tablet_bootstrap.cc:654] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:31.626552 3384 log.cc:824] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:31.627779 3385 tablet_bootstrap.cc:654] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:31.628494 3383 tablet_bootstrap.cc:492] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: No bootstrap required, opened a new log
I20260619 17:00:31.628820 3383 ts_tablet_manager.cc:1397] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 17:00:31.629168 3385 log.cc:824] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:31.630088 3384 tablet_bootstrap.cc:492] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: No bootstrap required, opened a new log
I20260619 17:00:31.630404 3384 ts_tablet_manager.cc:1397] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260619 17:00:31.632740 3385 tablet_bootstrap.cc:492] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: No bootstrap required, opened a new log
I20260619 17:00:31.633117 3385 ts_tablet_manager.cc:1397] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260619 17:00:31.637905 3383 raft_consensus.cc:348] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } }
I20260619 17:00:31.638409 3383 raft_consensus.cc:374] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:31.638517 3383 raft_consensus.cc:729] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3fca694e3c4a48944dde6d1d415b88, State: Initialized, Role: FOLLOWER
I20260619 17:00:31.639206 3383 consensus_queue.cc:260] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } }
I20260619 17:00:31.640511 3365 heartbeater.cc:503] Master 127.16.48.62:44267 was elected leader, sending a full tablet report...
I20260619 17:00:31.641278 3383 ts_tablet_manager.cc:1428] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 17:00:31.641721 3384 raft_consensus.cc:348] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } }
I20260619 17:00:31.642216 3384 raft_consensus.cc:374] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:31.642325 3384 raft_consensus.cc:729] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8900d129b99a4d25ba7a5ad10adff670, State: Initialized, Role: FOLLOWER
I20260619 17:00:31.642990 3384 consensus_queue.cc:260] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } }
I20260619 17:00:31.644743 3235 heartbeater.cc:503] Master 127.16.48.62:44267 was elected leader, sending a full tablet report...
I20260619 17:00:31.644614 3385 raft_consensus.cc:348] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } }
I20260619 17:00:31.645056 3385 raft_consensus.cc:374] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:31.645243 3385 raft_consensus.cc:729] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538d3e696ae54e31b5e134758513418e, State: Initialized, Role: FOLLOWER
I20260619 17:00:31.645530 3384 ts_tablet_manager.cc:1428] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260619 17:00:31.645896 3385 consensus_queue.cc:260] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } }
I20260619 17:00:31.647944 3385 ts_tablet_manager.cc:1428] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260619 17:00:31.755609 3366 tablet.cc:2369] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 17:00:31.759945 3098 tablet.cc:2369] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 17:00:31.764436 3236 tablet.cc:2369] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:31.937126 3389 raft_consensus.cc:482] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 17:00:31.937484 3389 raft_consensus.cc:504] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } }
I20260619 17:00:31.939392 3389 leader_election.cc:288] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8900d129b99a4d25ba7a5ad10adff670 (127.16.48.2:44099), 538d3e696ae54e31b5e134758513418e (127.16.48.1:45395)
I20260619 17:00:31.947058 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.939872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:53205 (local address 127.16.48.2:44099)
0619 17:00:31.940094 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:31.940099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:31.940781 (+ 682us) server_negotiation.cc:407] Connection header received
0619 17:00:31.941012 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:31.941024 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:31.941102 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:31.941221 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:31.941738 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.942742 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.945113 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.946110 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.946230 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.946303 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:31.946648 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 17:00:31.946748 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 17:00:31.947189 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.940461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:44099 (local address 127.16.48.3:53205)
0619 17:00:31.940666 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:31.940686 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:31.940861 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:31.941274 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:31.941279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:31.941307 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:31.941538 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.941552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.942890 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.942897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.944934 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.944949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.946281 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.946288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.946468 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.946505 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 17:00:31.946623 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 17:00:31.946784 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":33}
I20260619 17:00:31.947861 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.940206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:58617 (local address 127.16.48.1:45395)
0619 17:00:31.940374 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:31.940379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:31.941470 (+ 1091us) server_negotiation.cc:407] Connection header received
0619 17:00:31.941592 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:31.941600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:31.941702 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:31.941786 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:31.942408 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.943326 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.945671 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.946740 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.946885 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.947004 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:31.947397 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 17:00:31.947577 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 17:00:31.948532 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:31.940501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45395 (local address 127.16.48.3:58617)
0619 17:00:31.941355 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:31.941374 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:31.941487 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:31.941843 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:31.941848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:31.941875 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:31.942248 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.942261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.943489 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.943493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.945446 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:31.945461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:31.946916 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:31.946922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:31.947182 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:31.947225 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 17:00:31.948053 (+ 828us) client_negotiation.cc:240] Negotiation successful
0619 17:00:31.948251 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":37,"thread_start_us":500,"threads_started":1}
I20260619 17:00:31.948388 3190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7d9078c1d0469a9c43facd7b5f8190" candidate_uuid: "ed3fca694e3c4a48944dde6d1d415b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8900d129b99a4d25ba7a5ad10adff670" is_pre_election: true
I20260619 17:00:31.949026 3190 raft_consensus.cc:2454] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3fca694e3c4a48944dde6d1d415b88 in term 0.
I20260619 17:00:31.949716 3052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7d9078c1d0469a9c43facd7b5f8190" candidate_uuid: "ed3fca694e3c4a48944dde6d1d415b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538d3e696ae54e31b5e134758513418e" is_pre_election: true
I20260619 17:00:31.949837 3254 leader_election.cc:302] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8900d129b99a4d25ba7a5ad10adff670, ed3fca694e3c4a48944dde6d1d415b88; no voters:
I20260619 17:00:31.950299 3052 raft_consensus.cc:2454] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3fca694e3c4a48944dde6d1d415b88 in term 0.
I20260619 17:00:31.950328 3389 raft_consensus.cc:2781] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 17:00:31.950484 3389 raft_consensus.cc:482] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 17:00:31.950628 3389 raft_consensus.cc:3037] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:31.953315 3389 raft_consensus.cc:504] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } }
I20260619 17:00:31.953956 3389 leader_election.cc:288] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [CANDIDATE]: Term 1 election: Requested vote from peers 8900d129b99a4d25ba7a5ad10adff670 (127.16.48.2:44099), 538d3e696ae54e31b5e134758513418e (127.16.48.1:45395)
I20260619 17:00:31.954509 3190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7d9078c1d0469a9c43facd7b5f8190" candidate_uuid: "ed3fca694e3c4a48944dde6d1d415b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8900d129b99a4d25ba7a5ad10adff670"
I20260619 17:00:31.954519 3052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7d9078c1d0469a9c43facd7b5f8190" candidate_uuid: "ed3fca694e3c4a48944dde6d1d415b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538d3e696ae54e31b5e134758513418e"
I20260619 17:00:31.954744 3190 raft_consensus.cc:3037] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:31.954756 3052 raft_consensus.cc:3037] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:31.957123 3052 raft_consensus.cc:2454] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3fca694e3c4a48944dde6d1d415b88 in term 1.
I20260619 17:00:31.957134 3190 raft_consensus.cc:2454] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3fca694e3c4a48944dde6d1d415b88 in term 1.
I20260619 17:00:31.957669 3254 leader_election.cc:302] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8900d129b99a4d25ba7a5ad10adff670, ed3fca694e3c4a48944dde6d1d415b88; no voters:
I20260619 17:00:31.958029 3389 raft_consensus.cc:2781] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:31.958447 3389 raft_consensus.cc:686] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 1 LEADER]: Becoming Leader. State: Replica: ed3fca694e3c4a48944dde6d1d415b88, State: Running, Role: LEADER
I20260619 17:00:31.958962 3389 consensus_queue.cc:237] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } }
I20260619 17:00:31.964358 2911 catalog_manager.cc:5495] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3fca694e3c4a48944dde6d1d415b88 (127.16.48.3). New cstate: current_term: 1 leader_uuid: "ed3fca694e3c4a48944dde6d1d415b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed3fca694e3c4a48944dde6d1d415b88" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 40915 } health_report { overall_health: HEALTHY } } }
I20260619 17:00:32.064429 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.058469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.16.48.3:40915)
0619 17:00:32.058658 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:32.058662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:32.059026 (+ 364us) server_negotiation.cc:407] Connection header received
0619 17:00:32.059200 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:32.059203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:32.059294 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:32.059461 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:00:32.059786 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:32.060667 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:32.062074 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:32.062540 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:32.062646 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:32.063207 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:00:32.063772 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:00:32.063910 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:32.064048 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 17:00:32.064089 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 17:00:32.073501 3190 raft_consensus.cc:1261] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [term 1 FOLLOWER]: Refusing update from remote peer ed3fca694e3c4a48944dde6d1d415b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:32.073501 3052 raft_consensus.cc:1261] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [term 1 FOLLOWER]: Refusing update from remote peer ed3fca694e3c4a48944dde6d1d415b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:32.074700 3389 consensus_queue.cc:1035] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "538d3e696ae54e31b5e134758513418e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 45395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:32.075027 3393 consensus_queue.cc:1035] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8900d129b99a4d25ba7a5ad10adff670" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:32.104417 2910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:32.104743 2910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:32.108381 2910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6dfd4258e5142f5aa26be7941db4d80: Sending DeleteTablet for 3 replicas of tablet ae7d9078c1d0469a9c43facd7b5f8190
I20260619 17:00:32.109735 3032 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7d9078c1d0469a9c43facd7b5f8190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:32 UTC) from {username='slave'} at 127.0.0.1:56090
I20260619 17:00:32.109717 3170 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7d9078c1d0469a9c43facd7b5f8190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:32 UTC) from {username='slave'} at 127.0.0.1:49634
I20260619 17:00:32.109740 3300 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7d9078c1d0469a9c43facd7b5f8190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:32 UTC) from {username='slave'} at 127.0.0.1:45146
I20260619 17:00:32.115439 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 2970
I20260619 17:00:32.123495 3408 tablet_replica.cc:330] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: stopping tablet replica
I20260619 17:00:32.124593 3409 tablet_replica.cc:330] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: stopping tablet replica
I20260619 17:00:32.125113 3407 tablet_replica.cc:330] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: stopping tablet replica
I20260619 17:00:32.125170 3409 raft_consensus.cc:2229] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:32.125718 3407 raft_consensus.cc:2229] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:32.125793 3409 raft_consensus.cc:2258] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:32.126343 3407 raft_consensus.cc:2258] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:32.126431 3408 raft_consensus.cc:2229] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 1 LEADER]: Raft consensus shutting down.
I20260619 17:00:32.127431 3408 raft_consensus.cc:2258] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:32.129148 3407 ts_tablet_manager.cc:1905] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:32.131471 3409 ts_tablet_manager.cc:1905] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:32.131976 3408 ts_tablet_manager.cc:1905] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:32.137857 3407 ts_tablet_manager.cc:1918] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:32.138085 3407 log.cc:1196] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/ae7d9078c1d0469a9c43facd7b5f8190
I20260619 17:00:32.138643 3407 ts_tablet_manager.cc:1939] T ae7d9078c1d0469a9c43facd7b5f8190 P 8900d129b99a4d25ba7a5ad10adff670: Deleting consensus metadata
I20260619 17:00:32.139703 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.116619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.16.48.1:45395)
0619 17:00:32.116819 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:32.116825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:32.121015 (+ 4190us) server_negotiation.cc:407] Connection header received
0619 17:00:32.121272 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:32.121281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:32.121434 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:32.121551 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:32.125507 (+ 3956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:32.126972 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:32.129159 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:32.129890 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:32.130014 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:32.137401 (+ 7387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:32.137418 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:32.137441 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:32.138324 (+ 883us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:32.138398 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:32.138430 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:32.138560 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:32.139062 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 17:00:32.139209 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 17:00:32.139361 3409 ts_tablet_manager.cc:1918] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:32.140431 3409 log.cc:1196] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/ae7d9078c1d0469a9c43facd7b5f8190
I20260619 17:00:32.140550 3092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:32.140938 2896 catalog_manager.cc:4841] TS 8900d129b99a4d25ba7a5ad10adff670 (127.16.48.2:44099): tablet ae7d9078c1d0469a9c43facd7b5f8190 (table test-table [id=56f9c9a1abd64c09b286d4c8d7dcf644]) successfully deleted
I20260619 17:00:32.143563 3408 ts_tablet_manager.cc:1918] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:32.143810 3408 log.cc:1196] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/ae7d9078c1d0469a9c43facd7b5f8190
I20260619 17:00:32.144556 3408 ts_tablet_manager.cc:1939] T ae7d9078c1d0469a9c43facd7b5f8190 P ed3fca694e3c4a48944dde6d1d415b88: Deleting consensus metadata
I20260619 17:00:32.146906 2898 catalog_manager.cc:4841] TS ed3fca694e3c4a48944dde6d1d415b88 (127.16.48.3:40915): tablet ae7d9078c1d0469a9c43facd7b5f8190 (table test-table [id=56f9c9a1abd64c09b286d4c8d7dcf644]) successfully deleted
I20260619 17:00:32.237092 3409 ts_tablet_manager.cc:1939] T ae7d9078c1d0469a9c43facd7b5f8190 P 538d3e696ae54e31b5e134758513418e: Deleting consensus metadata
I20260619 17:00:32.238086 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 2970
I20260619 17:00:32.259035 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 3101
W20260619 17:00:32.259037 2897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.48.1:45395 (error 108)
I20260619 17:00:32.261345 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.260541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45395 (local address 127.0.0.1:56108)
0619 17:00:32.260954 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:32.261060 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":206,"threads_started":1}
W20260619 17:00:32.261736 2897 catalog_manager.cc:4584] TS 538d3e696ae54e31b5e134758513418e (127.16.48.1:45395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae7d9078c1d0469a9c43facd7b5f8190: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45395: connect: Connection refused (error 111)
I20260619 17:00:32.268625 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.260269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.16.48.2:44099)
0619 17:00:32.260461 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:32.260466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:32.260486 (+ 20us) server_negotiation.cc:407] Connection header received
0619 17:00:32.261200 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:32.261207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:32.261343 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:32.261473 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:32.262500 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:32.263332 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:32.265021 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:32.265581 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:32.265739 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:32.267002 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:32.267014 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:32.267031 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:32.267693 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:32.267749 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:32.267771 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:32.267929 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:32.268082 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 17:00:32.268235 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 17:00:32.269436 3230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:32.315517 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.314933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45395 (local address 127.0.0.1:56120)
0619 17:00:32.315218 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:32.315328 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 17:00:32.316867 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.316395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45395 (local address 127.0.0.1:56136)
0619 17:00:32.316592 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:32.316683 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 17:00:32.364530 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 3101
I20260619 17:00:32.377916 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.377254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45395 (local address 127.0.0.1:56138)
0619 17:00:32.377565 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:32.377696 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 17:00:32.379331 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.378822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45395 (local address 127.0.0.1:56152)
0619 17:00:32.379029 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:32.379135 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 17:00:32.384014 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 3238
I20260619 17:00:32.392014 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.384898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.16.48.3:40915)
0619 17:00:32.385077 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:32.385085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:32.385198 (+ 113us) server_negotiation.cc:407] Connection header received
0619 17:00:32.385428 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:32.385432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:32.385529 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:32.385648 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:32.386249 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:32.387093 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:32.388726 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:32.389191 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:32.389344 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:32.390521 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:32.390530 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:32.390545 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:32.391187 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:32.391237 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:32.391258 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:32.391405 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:32.391543 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 17:00:32.391650 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 17:00:32.392833 3360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:32.460700 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.460197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45395 (local address 127.0.0.1:56158)
0619 17:00:32.460423 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:32.460542 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 17:00:32.461995 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:32.461513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:45395 (local address 127.0.0.1:56166)
0619 17:00:32.461752 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:32.461832 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 17:00:32.518496 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 3238
I20260619 17:00:32.538591 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 2880
I20260619 17:00:32.539821 2941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:32.651324 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 2880
2026-06-19T17:00:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3305 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:32 dist-test-slave-3ffd krb5kdc[3419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:32 dist-test-slave-3ffd krb5kdc[3419](info): set up 2 sockets
Jun 19 17:00:32 dist-test-slave-3ffd krb5kdc[3419](info): commencing operation
krb5kdc: starting...
W20260619 17:00:33.784845 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.005s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:33 dist-test-slave-3ffd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:00:34.064395 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36571
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:34.426488 3433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:34.426828 3433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:34.426923 3433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:34.434834 3433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:34.434976 3433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:34.435032 3433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:34.435091 3433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:34.435137 3433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:34.445562 3433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:36571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:34.446940 3433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:34.448356 3433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:34.461529 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:34.461670 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:34.461724 3446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:34.462177 3433 server_base.cc:1034] running on GCE node
I20260619 17:00:34.463109 3433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:34.464794 3433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:34.466086 3433 hybrid_clock.cc:648] HybridClock initialized: now 1781888434466026 us; error 75 us; skew 500 ppm
I20260619 17:00:34.466497 3433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:34 dist-test-slave-3ffd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888434, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:34.470196 3433 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:00:34.472024 3433 webserver.cc:460] Webserver started at http://127.16.48.62:37135/ using document root <none> and password file <none>
I20260619 17:00:34.472851 3433 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:34.473016 3433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:34.473352 3433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:34.476801 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "486b4832935f4c19866fe8a3bd9dc57d"
format_stamp: "Formatted at 2026-06-19 17:00:34 on dist-test-slave-3ffd"
server_key: "1d108e6a5d92e454c2cdf0b527daa3a3"
server_key_iv: "b2b87a7f9044938d6294eb7891198f42"
server_key_version: "encryptionkey@0"
I20260619 17:00:34.477696 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "486b4832935f4c19866fe8a3bd9dc57d"
format_stamp: "Formatted at 2026-06-19 17:00:34 on dist-test-slave-3ffd"
server_key: "1d108e6a5d92e454c2cdf0b527daa3a3"
server_key_iv: "b2b87a7f9044938d6294eb7891198f42"
server_key_version: "encryptionkey@0"
I20260619 17:00:34.483644 3433 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 17:00:34.487598 3453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:34.489212 3433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 17:00:34.489436 3433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "486b4832935f4c19866fe8a3bd9dc57d"
format_stamp: "Formatted at 2026-06-19 17:00:34 on dist-test-slave-3ffd"
server_key: "1d108e6a5d92e454c2cdf0b527daa3a3"
server_key_iv: "b2b87a7f9044938d6294eb7891198f42"
server_key_version: "encryptionkey@0"
I20260619 17:00:34.489672 3433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:34.509936 3433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:34.514742 3433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:34.515064 3433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:34.535683 3433 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:36571
I20260619 17:00:34.535734 3504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:36571 every 8 connection(s)
I20260619 17:00:34.537645 3433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:34.542331 3505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:34.542331 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 3433
I20260619 17:00:34.542554 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:34.543162 16576 external_mini_cluster.cc:1428] Setting key 373aa44077b8ce7ee8e7da9f0df08989
I20260619 17:00:34.559387 3505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d: Bootstrap starting.
Jun 19 17:00:34 dist-test-slave-3ffd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:34.563246 3505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:34.564702 3505 log.cc:824] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:34.567940 3505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d: No bootstrap required, opened a new log
I20260619 17:00:34.570103 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:34.545354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.16.48.62:36571)
0619 17:00:34.546348 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:34.546363 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:34.546418 (+ 55us) server_negotiation.cc:407] Connection header received
0619 17:00:34.548473 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:34.548503 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:34.549034 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:34.549501 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:34.551583 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:34.553202 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:34.555240 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:34.556261 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:34.556462 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:34.561841 (+ 5379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:34.561851 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:34.561869 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:34.561934 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:34.565662 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:34.566504 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:34.566518 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:34.566526 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:34.566716 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:34.567135 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:34.567141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:34.567144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:34.567697 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:34.568044 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:34.568571 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 17:00:34.569044 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":462,"threads_started":1}
I20260619 17:00:34.577188 3505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486b4832935f4c19866fe8a3bd9dc57d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36571 } }
I20260619 17:00:34.577679 3505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:34.577804 3505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486b4832935f4c19866fe8a3bd9dc57d, State: Initialized, Role: FOLLOWER
I20260619 17:00:34.578522 3505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486b4832935f4c19866fe8a3bd9dc57d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36571 } }
I20260619 17:00:34.578859 3505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:34.579027 3505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:34.579244 3505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:34.581689 3505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486b4832935f4c19866fe8a3bd9dc57d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36571 } }
I20260619 17:00:34.582235 3505 leader_election.cc:302] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486b4832935f4c19866fe8a3bd9dc57d; no voters:
I20260619 17:00:34.582814 3505 leader_election.cc:288] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:34.582995 3510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:34.583752 3510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [term 1 LEADER]: Becoming Leader. State: Replica: 486b4832935f4c19866fe8a3bd9dc57d, State: Running, Role: LEADER
I20260619 17:00:34.584410 3510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486b4832935f4c19866fe8a3bd9dc57d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36571 } }
I20260619 17:00:34.585316 3505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:34.587682 3511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486b4832935f4c19866fe8a3bd9dc57d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486b4832935f4c19866fe8a3bd9dc57d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36571 } } }
I20260619 17:00:34.587673 3512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486b4832935f4c19866fe8a3bd9dc57d. Latest consensus state: current_term: 1 leader_uuid: "486b4832935f4c19866fe8a3bd9dc57d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486b4832935f4c19866fe8a3bd9dc57d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 36571 } } }
I20260619 17:00:34.588420 3512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:34.588420 3511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:34.590764 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:34.597512 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:34.607913 3516 catalog_manager.cc:1347] Generated new cluster ID: 3aba58993bcd4e228b9582f97add6b24
I20260619 17:00:34.608062 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:34.621016 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:34.623077 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:34.630422 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d: Generated new TSK 0
I20260619 17:00:34.631583 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
Entry for principal kudu/127.16.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab.
I20260619 17:00:34.701947 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36571
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:35.088436 3531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:35.088799 3531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:35.088969 3531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:35.097350 3531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:35.097548 3531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:35.097733 3531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 17:00:35.108556 3531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.1.keytab
--principal=kudu/127.16.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:35.110005 3531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:35.111413 3531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:35.126020 3539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:35.125957 3537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:35.125998 3536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:35.127203 3531 server_base.cc:1034] running on GCE node
I20260619 17:00:35.127985 3531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:35.129134 3531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:35.130411 3531 hybrid_clock.cc:648] HybridClock initialized: now 1781888435130359 us; error 53 us; skew 500 ppm
I20260619 17:00:35.130785 3531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:35 dist-test-slave-3ffd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:35.134384 3531 init.cc:377] Logged in from keytab as kudu/127.16.48.1@KRBTEST.COM (short username kudu)
I20260619 17:00:35.136168 3531 webserver.cc:460] Webserver started at http://127.16.48.1:44303/ using document root <none> and password file <none>
I20260619 17:00:35.136960 3531 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:35.137133 3531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:35.137485 3531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:35.140815 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "1d85b2662b94459f8f1a084e55637ce6"
format_stamp: "Formatted at 2026-06-19 17:00:35 on dist-test-slave-3ffd"
server_key: "7892dff1692160c7eb09a3907e069054"
server_key_iv: "65b776badcd8ad6d327d5bda67a20f5c"
server_key_version: "encryptionkey@0"
I20260619 17:00:35.141741 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "1d85b2662b94459f8f1a084e55637ce6"
format_stamp: "Formatted at 2026-06-19 17:00:35 on dist-test-slave-3ffd"
server_key: "7892dff1692160c7eb09a3907e069054"
server_key_iv: "65b776badcd8ad6d327d5bda67a20f5c"
server_key_version: "encryptionkey@0"
I20260619 17:00:35.147892 3531 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 17:00:35.152194 3546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:35.153815 3531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 17:00:35.154049 3531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "1d85b2662b94459f8f1a084e55637ce6"
format_stamp: "Formatted at 2026-06-19 17:00:35 on dist-test-slave-3ffd"
server_key: "7892dff1692160c7eb09a3907e069054"
server_key_iv: "65b776badcd8ad6d327d5bda67a20f5c"
server_key_version: "encryptionkey@0"
I20260619 17:00:35.154301 3531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:35.168529 3531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:35.173751 3531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:35.174117 3531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:35.175220 3531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:35.176883 3531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:35.177001 3531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:35.177153 3531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:35.177232 3531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:35.211972 3531 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:35793
I20260619 17:00:35.212033 3658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:35793 every 8 connection(s)
I20260619 17:00:35.213510 3531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 17:00:35.214020 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 3531
I20260619 17:00:35.214255 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 17:00:35.214807 16576 external_mini_cluster.cc:1428] Setting key 52b8f5db430b4aedc12389ba542cba7e
Jun 19 17:00:35 dist-test-slave-3ffd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.1@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:35.239290 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:35.216549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:59283 (local address 127.16.48.62:36571)
0619 17:00:35.216975 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:35.216979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:35.218403 (+ 1424us) server_negotiation.cc:407] Connection header received
0619 17:00:35.221476 (+ 3073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:35.221481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:35.221575 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:35.221704 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:35.224064 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:35.224912 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:35.227784 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:35.228299 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:35.228452 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:35.232803 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:35.232809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:35.232813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:35.232851 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:35.235609 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:35.236761 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:35.236765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:35.236768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:35.236903 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:35.237436 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:35.237440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:35.237442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:35.237715 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:35.237828 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:35.238822 (+ 994us) server_negotiation.cc:299] Negotiation successful
0619 17:00:35.239015 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":208,"threads_started":1}
I20260619 17:00:35.241091 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:35.217281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36571 (local address 127.16.48.1:59283)
0619 17:00:35.218205 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:35.218262 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:35.221110 (+ 2848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:35.222017 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:35.222032 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:35.222908 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:35.223816 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:35.223847 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:35.225104 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:35.225113 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:35.227543 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:35.227563 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:35.228572 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:35.228588 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:35.228855 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:35.229617 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:00:35.229643 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:35.232500 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:35.235799 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:35.235814 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:35.235837 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:35.236629 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:35.237021 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:35.237029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:35.237035 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:35.237317 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:35.237857 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:35.237868 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:35.238384 (+ 516us) client_negotiation.cc:769] Sending connection context
0619 17:00:35.238727 (+ 343us) client_negotiation.cc:240] Negotiation successful
0619 17:00:35.239409 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":340,"threads_started":1}
I20260619 17:00:35.242928 3659 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36571
I20260619 17:00:35.243284 3659 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:35.244086 3659 heartbeater.cc:511] Master 127.16.48.62:36571 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.48.2@KRBTEST.COM; defaulting to no policy
I20260619 17:00:35.246615 3470 ts_manager.cc:194] Registered new tserver with Master: 1d85b2662b94459f8f1a084e55637ce6 (127.16.48.1:35793)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.2@KRBTEST.COM" created.
I20260619 17:00:35.248462 3470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.1@KRBTEST.COM'} at 127.16.48.1:59283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
Entry for principal kudu/127.16.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab.
I20260619 17:00:35.280726 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36571
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:35.639850 3665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:35.640288 3665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:35.640456 3665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:35.648429 3665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:35.648594 3665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:35.648788 3665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 17:00:35.659325 3665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.2.keytab
--principal=kudu/127.16.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:35.660866 3665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:35.662272 3665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:35.676623 3670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:35.676668 3671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:35.676841 3673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:35.677443 3665 server_base.cc:1034] running on GCE node
I20260619 17:00:35.678196 3665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:35.679301 3665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:35.680544 3665 hybrid_clock.cc:648] HybridClock initialized: now 1781888435680470 us; error 76 us; skew 500 ppm
I20260619 17:00:35.680907 3665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:35 dist-test-slave-3ffd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:35.684468 3665 init.cc:377] Logged in from keytab as kudu/127.16.48.2@KRBTEST.COM (short username kudu)
I20260619 17:00:35.686280 3665 webserver.cc:460] Webserver started at http://127.16.48.2:40971/ using document root <none> and password file <none>
I20260619 17:00:35.687079 3665 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:35.687235 3665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:35.687582 3665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:35.690964 3665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "130ad2350d644837a5f9c4cca58910df"
format_stamp: "Formatted at 2026-06-19 17:00:35 on dist-test-slave-3ffd"
server_key: "7df336850545e69e17edb6359df71d32"
server_key_iv: "16b3832fa386a20c0469a08328716ca4"
server_key_version: "encryptionkey@0"
I20260619 17:00:35.691854 3665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "130ad2350d644837a5f9c4cca58910df"
format_stamp: "Formatted at 2026-06-19 17:00:35 on dist-test-slave-3ffd"
server_key: "7df336850545e69e17edb6359df71d32"
server_key_iv: "16b3832fa386a20c0469a08328716ca4"
server_key_version: "encryptionkey@0"
I20260619 17:00:35.698181 3665 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 17:00:35.702338 3680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:35.703979 3665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 17:00:35.704231 3665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "130ad2350d644837a5f9c4cca58910df"
format_stamp: "Formatted at 2026-06-19 17:00:35 on dist-test-slave-3ffd"
server_key: "7df336850545e69e17edb6359df71d32"
server_key_iv: "16b3832fa386a20c0469a08328716ca4"
server_key_version: "encryptionkey@0"
I20260619 17:00:35.704478 3665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:35.770341 3665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:35.775650 3665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:35.775985 3665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:35.777086 3665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:35.778702 3665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:35.778805 3665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:35.778970 3665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:35.779045 3665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:35.815385 3665 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:38871
I20260619 17:00:35.815460 3792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:38871 every 8 connection(s)
I20260619 17:00:35.816898 3665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 17:00:35.819697 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 3665
I20260619 17:00:35.819921 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 17:00:35.820550 16576 external_mini_cluster.cc:1428] Setting key 57d91caf2f6fccb43dc79c1fb7dd3718
Jun 19 17:00:35 dist-test-slave-3ffd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.2@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:35.843590 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:35.819922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:46057 (local address 127.16.48.62:36571)
0619 17:00:35.820514 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:35.820521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:35.821719 (+ 1198us) server_negotiation.cc:407] Connection header received
0619 17:00:35.824447 (+ 2728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:35.824452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:35.824526 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:35.824627 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:35.827127 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:35.827926 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:35.830796 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:35.831339 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:35.831453 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:35.835679 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:35.835689 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:35.835696 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:35.835749 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:35.839674 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:35.840749 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:35.840758 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:35.840764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:35.840949 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:35.841405 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:35.841413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:35.841419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:35.841858 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:35.842026 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:35.842807 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 17:00:35.843078 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":288,"threads_started":1}
I20260619 17:00:35.844405 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:35.820573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36571 (local address 127.16.48.2:46057)
0619 17:00:35.821514 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:35.821570 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:35.824043 (+ 2473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:35.824967 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:35.824984 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:35.825932 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:35.826860 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:35.826892 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:35.828172 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:35.828181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:35.830569 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:35.830591 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:35.831633 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:35.831643 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:35.832016 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:35.832910 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:00:35.832952 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:35.835360 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:35.839901 (+ 4541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:35.839911 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:35.839928 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:35.840570 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:35.841072 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:35.841075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:35.841078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:35.841274 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:35.842007 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:35.842013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:35.842379 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 17:00:35.842646 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 17:00:35.843110 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":334,"threads_started":1}
I20260619 17:00:35.846143 3793 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36571
I20260619 17:00:35.846489 3793 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:35.847380 3793 heartbeater.cc:511] Master 127.16.48.62:36571 requested a full tablet report, sending...
I20260619 17:00:35.849504 3470 ts_manager.cc:194] Registered new tserver with Master: 130ad2350d644837a5f9c4cca58910df (127.16.48.2:38871)
I20260619 17:00:35.850833 3470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.2@KRBTEST.COM'} at 127.16.48.2:46057
WARNING: no policy specified for kudu/127.16.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
Entry for principal kudu/127.16.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab.
I20260619 17:00:35.887310 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36571
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:36.243698 3799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:36.244076 3799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:36.244270 3799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:36.251999 3799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:36.252238 3799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260619 17:00:36.252101 3659 heartbeater.cc:503] Master 127.16.48.62:36571 was elected leader, sending a full tablet report...
W20260619 17:00:36.252437 3799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 17:00:36.263657 3799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.3.keytab
--principal=kudu/127.16.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:36.265192 3799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:36.266665 3799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:36.281484 3807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:36.281605 3805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:36.281517 3804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:36.282400 3799 server_base.cc:1034] running on GCE node
I20260619 17:00:36.283340 3799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:36.285449 3799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:36.286844 3799 hybrid_clock.cc:648] HybridClock initialized: now 1781888436286775 us; error 60 us; skew 500 ppm
I20260619 17:00:36.287336 3799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:36 dist-test-slave-3ffd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888436, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:36.292025 3799 init.cc:377] Logged in from keytab as kudu/127.16.48.3@KRBTEST.COM (short username kudu)
I20260619 17:00:36.308101 3799 webserver.cc:460] Webserver started at http://127.16.48.3:34007/ using document root <none> and password file <none>
I20260619 17:00:36.309000 3799 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:36.309180 3799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:36.309558 3799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:36.312934 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "099c9c0b347544fdb0dcd7677fa7da29"
format_stamp: "Formatted at 2026-06-19 17:00:36 on dist-test-slave-3ffd"
server_key: "809216ebb24281b34822f6ce6cc55b5b"
server_key_iv: "f5b132747d13e1ab5c1211e272d77918"
server_key_version: "encryptionkey@0"
I20260619 17:00:36.313827 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "099c9c0b347544fdb0dcd7677fa7da29"
format_stamp: "Formatted at 2026-06-19 17:00:36 on dist-test-slave-3ffd"
server_key: "809216ebb24281b34822f6ce6cc55b5b"
server_key_iv: "f5b132747d13e1ab5c1211e272d77918"
server_key_version: "encryptionkey@0"
I20260619 17:00:36.320292 3799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 17:00:36.324767 3814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:36.326656 3799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260619 17:00:36.326905 3799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "099c9c0b347544fdb0dcd7677fa7da29"
format_stamp: "Formatted at 2026-06-19 17:00:36 on dist-test-slave-3ffd"
server_key: "809216ebb24281b34822f6ce6cc55b5b"
server_key_iv: "f5b132747d13e1ab5c1211e272d77918"
server_key_version: "encryptionkey@0"
I20260619 17:00:36.327191 3799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:36.351879 3799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:36.357255 3799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:36.357615 3799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:36.358867 3799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:36.360666 3799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:36.360782 3799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:36.360940 3799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:36.361014 3799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:36.402395 3799 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:35819
I20260619 17:00:36.402427 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:35819 every 8 connection(s)
I20260619 17:00:36.403841 3799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 17:00:36.406080 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 3799
I20260619 17:00:36.406281 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 17:00:36.406872 16576 external_mini_cluster.cc:1428] Setting key aab83cc19868ab996208dce446ef7171
Jun 19 17:00:36 dist-test-slave-3ffd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888436, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.3@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:00:36.431684 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.407152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:38169 (local address 127.16.48.62:36571)
0619 17:00:36.412617 (+ 5465us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:36.412626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:36.412650 (+ 24us) server_negotiation.cc:407] Connection header received
0619 17:00:36.412717 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:36.412724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:36.412825 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:36.412957 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:36.414593 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.416079 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.417872 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.418746 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.418913 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.423081 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:36.423088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:36.423095 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:36.423167 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:36.427620 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:36.428633 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:36.428642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:36.428648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:36.428832 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:36.429313 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:36.429320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:36.429326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:36.429776 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:36.429967 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:36.430764 (+ 797us) server_negotiation.cc:299] Negotiation successful
0619 17:00:36.431082 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5292,"thread_start_us":5131,"threads_started":1}
I20260619 17:00:36.432531 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.407803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:36571 (local address 127.16.48.3:38169)
0619 17:00:36.408818 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:36.408876 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:36.412141 (+ 3265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:36.413188 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:36.413197 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:36.413778 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:36.414382 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.414404 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.416278 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.416282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.417678 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.417692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.419002 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.419007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.419260 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.420088 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:00:36.420184 (+ 96us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:36.422782 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:36.427838 (+ 5056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:36.427848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:36.427865 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:36.428483 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:36.428969 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:00:36.428973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:00:36.428975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:00:36.429184 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:00:36.429932 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:36.429940 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:36.430324 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 17:00:36.430586 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 17:00:36.431051 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":612,"threads_started":1}
I20260619 17:00:36.434130 3927 heartbeater.cc:347] Connected to a master server at 127.16.48.62:36571
I20260619 17:00:36.434480 3927 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:36.435420 3927 heartbeater.cc:511] Master 127.16.48.62:36571 requested a full tablet report, sending...
I20260619 17:00:36.437927 3469 ts_manager.cc:194] Registered new tserver with Master: 099c9c0b347544fdb0dcd7677fa7da29 (127.16.48.3:35819)
I20260619 17:00:36.438992 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.48.3@KRBTEST.COM'} at 127.16.48.3:38169
I20260619 17:00:36.451344 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 17:00:36.475847 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.457679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.16.48.62:36571)
0619 17:00:36.457934 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:36.457944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:36.458508 (+ 564us) server_negotiation.cc:407] Connection header received
0619 17:00:36.458717 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:36.458726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:36.458846 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:36.459007 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:36.460831 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.462268 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.463762 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.464610 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.464774 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.467457 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:36.467466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:36.467474 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:36.467531 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:36.472085 (+ 4554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:36.473000 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:36.473009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:36.473015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:36.473213 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:36.473635 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:36.473643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:36.473649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:36.474211 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:36.474513 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:36.475042 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 17:00:36.475363 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 17:00:36.480834 3469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 17:00:36.488874 3469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 17:00:36.532979 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.504795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:35793 (local address 127.0.0.1:34194)
0619 17:00:36.505629 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:36.505680 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:36.505847 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:36.514400 (+ 8553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:36.514414 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:36.514473 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:36.515090 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.515113 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.518785 (+ 3672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.518795 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.524806 (+ 6011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.524842 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.531481 (+ 6639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.531491 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.531840 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.531953 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 17:00:36.532147 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 17:00:36.532469 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":1558,"threads_started":1}
I20260619 17:00:36.535657 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.505383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.16.48.1:35793)
0619 17:00:36.506082 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:36.506094 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:36.506132 (+ 38us) server_negotiation.cc:407] Connection header received
0619 17:00:36.506248 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:36.506259 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:36.513448 (+ 7189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:36.513857 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:36.516306 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.518270 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.525184 (+ 6914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.527319 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.527527 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.527850 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:36.532706 (+ 4856us) server_negotiation.cc:299] Negotiation successful
0619 17:00:36.535271 (+ 2565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":766,"threads_started":1}
I20260619 17:00:36.540584 3594 tablet_service.cc:1467] Processing CreateTablet for tablet afd8ccaebfc142d2889b2005d4b5b40d (DEFAULT_TABLE table=test-table [id=79ecc354ddd24bbf948cf3bdd3eea787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:36.542641 3594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd8ccaebfc142d2889b2005d4b5b40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:36.543772 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.507196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:38871 (local address 127.0.0.1:58722)
0619 17:00:36.529785 (+ 22589us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:36.529809 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:36.530001 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:36.532258 (+ 2257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:36.532266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:36.532318 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:36.533412 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.533437 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.536625 (+ 3188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.536631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.539040 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.539058 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.542860 (+ 3802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.542867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.543083 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.543123 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 17:00:36.543210 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 17:00:36.543366 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22402,"thread_start_us":22802,"threads_started":1}
I20260619 17:00:36.544947 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.505808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:35819 (local address 127.0.0.1:38842)
0619 17:00:36.529609 (+ 23801us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:36.529641 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:36.529863 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:36.536944 (+ 7081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:36.536951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:36.536984 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:36.537479 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.537494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.539616 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.539624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.541858 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.541875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.544146 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.544153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.544329 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.544361 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 17:00:36.544440 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 17:00:36.544598 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23557,"thread_start_us":24448,"threads_started":1}
I20260619 17:00:36.545810 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.506432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.16.48.3:35819)
0619 17:00:36.534200 (+ 27768us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:36.534207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:36.534237 (+ 30us) server_negotiation.cc:407] Connection header received
0619 17:00:36.534320 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:36.534326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:36.534633 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:36.534860 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:36.537860 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.539420 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.542069 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.543583 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.543712 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.543970 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:36.545241 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0619 17:00:36.545427 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27586,"thread_start_us":27402,"threads_started":1}
I20260619 17:00:36.546262 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.507473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.16.48.2:38871)
0619 17:00:36.528764 (+ 21291us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:36.528776 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:36.530294 (+ 1518us) server_negotiation.cc:407] Connection header received
0619 17:00:36.530437 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:36.530450 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:36.530802 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:36.531069 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:36.533980 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.536435 (+ 2455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.540739 (+ 4304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.542338 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.542471 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.542699 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:36.544156 (+ 1457us) server_negotiation.cc:299] Negotiation successful
0619 17:00:36.545892 (+ 1736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21049,"thread_start_us":21297,"threads_started":1}
I20260619 17:00:36.549719 3862 tablet_service.cc:1467] Processing CreateTablet for tablet afd8ccaebfc142d2889b2005d4b5b40d (DEFAULT_TABLE table=test-table [id=79ecc354ddd24bbf948cf3bdd3eea787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:36.550235 3728 tablet_service.cc:1467] Processing CreateTablet for tablet afd8ccaebfc142d2889b2005d4b5b40d (DEFAULT_TABLE table=test-table [id=79ecc354ddd24bbf948cf3bdd3eea787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:36.551318 3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd8ccaebfc142d2889b2005d4b5b40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:36.551843 3728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd8ccaebfc142d2889b2005d4b5b40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:36.570051 3946 tablet_bootstrap.cc:492] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: Bootstrap starting.
I20260619 17:00:36.572531 3947 tablet_bootstrap.cc:492] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: Bootstrap starting.
I20260619 17:00:36.574374 3946 tablet_bootstrap.cc:654] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:36.574409 3948 tablet_bootstrap.cc:492] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: Bootstrap starting.
I20260619 17:00:36.576030 3946 log.cc:824] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:36.576843 3947 tablet_bootstrap.cc:654] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:36.578806 3947 log.cc:824] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:36.582094 3948 tablet_bootstrap.cc:654] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:36.583837 3948 log.cc:824] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:36.586902 3946 tablet_bootstrap.cc:492] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: No bootstrap required, opened a new log
I20260619 17:00:36.587371 3946 ts_tablet_manager.cc:1397] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20260619 17:00:36.589777 3947 tablet_bootstrap.cc:492] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: No bootstrap required, opened a new log
I20260619 17:00:36.590268 3947 ts_tablet_manager.cc:1397] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20260619 17:00:36.599210 3946 raft_consensus.cc:348] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } }
I20260619 17:00:36.600409 3946 raft_consensus.cc:374] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:36.600643 3946 raft_consensus.cc:729] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d85b2662b94459f8f1a084e55637ce6, State: Initialized, Role: FOLLOWER
I20260619 17:00:36.601662 3946 consensus_queue.cc:260] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } }
I20260619 17:00:36.602435 3947 raft_consensus.cc:348] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } }
I20260619 17:00:36.603029 3947 raft_consensus.cc:374] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:36.603189 3947 raft_consensus.cc:729] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099c9c0b347544fdb0dcd7677fa7da29, State: Initialized, Role: FOLLOWER
I20260619 17:00:36.603621 3948 tablet_bootstrap.cc:492] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: No bootstrap required, opened a new log
I20260619 17:00:36.604001 3948 ts_tablet_manager.cc:1397] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.000s
I20260619 17:00:36.604146 3947 consensus_queue.cc:260] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } }
I20260619 17:00:36.609722 3947 ts_tablet_manager.cc:1428] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260619 17:00:36.610517 3927 heartbeater.cc:503] Master 127.16.48.62:36571 was elected leader, sending a full tablet report...
I20260619 17:00:36.616110 3948 raft_consensus.cc:348] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } }
I20260619 17:00:36.616837 3948 raft_consensus.cc:374] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:36.616995 3948 raft_consensus.cc:729] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130ad2350d644837a5f9c4cca58910df, State: Initialized, Role: FOLLOWER
I20260619 17:00:36.617897 3948 consensus_queue.cc:260] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } }
I20260619 17:00:36.619571 3946 ts_tablet_manager.cc:1428] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: Time spent starting tablet: real 0.032s user 0.014s sys 0.014s
I20260619 17:00:36.620944 3793 heartbeater.cc:503] Master 127.16.48.62:36571 was elected leader, sending a full tablet report...
I20260619 17:00:36.621490 3948 ts_tablet_manager.cc:1428] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260619 17:00:36.655840 3928 tablet.cc:2369] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 17:00:36.716606 3660 tablet.cc:2369] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:36.759199 3952 raft_consensus.cc:482] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 17:00:36.759543 3952 raft_consensus.cc:504] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } }
I20260619 17:00:36.761406 3952 leader_election.cc:288] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 130ad2350d644837a5f9c4cca58910df (127.16.48.2:38871), 099c9c0b347544fdb0dcd7677fa7da29 (127.16.48.3:35819)
I20260619 17:00:36.768873 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.762149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:55345 (local address 127.16.48.3:35819)
0619 17:00:36.762349 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:36.762356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:36.762875 (+ 519us) server_negotiation.cc:407] Connection header received
0619 17:00:36.763059 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:36.763062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:36.763126 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:36.763237 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:36.763958 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.764752 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.766810 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.767831 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.767968 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.768099 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:36.768459 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 17:00:36.768593 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 17:00:36.768999 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.761834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:35819 (local address 127.16.48.1:55345)
0619 17:00:36.762748 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:36.762763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:36.762895 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:36.763282 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:36.763287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:36.763312 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:36.763790 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.763807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.764923 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.764927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.766600 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.766620 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.768077 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.768086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.768279 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.768307 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 17:00:36.768431 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 17:00:36.768593 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":44,"thread_start_us":531,"threads_started":1}
I20260619 17:00:36.769167 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.761763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:42287 (local address 127.16.48.2:38871)
0619 17:00:36.762030 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:36.762039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:36.762727 (+ 688us) server_negotiation.cc:407] Connection header received
0619 17:00:36.763020 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:36.763027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:36.763105 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:36.763175 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:36.763861 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.764731 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.766859 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.767829 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.767984 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.768084 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:36.768720 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 17:00:36.768874 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 17:00:36.769943 3956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.761702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:38871 (local address 127.16.48.1:42287)
0619 17:00:36.762562 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:36.762600 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:36.762818 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:36.763229 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:36.763234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:36.763258 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:36.763698 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.763711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.764872 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.764878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.766638 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:36.766652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.768279 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.768286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:36.768516 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.768551 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 17:00:36.769338 (+ 787us) client_negotiation.cc:240] Negotiation successful
0619 17:00:36.769526 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":72,"thread_start_us":737,"threads_started":1}
I20260619 17:00:36.770257 3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd8ccaebfc142d2889b2005d4b5b40d" candidate_uuid: "1d85b2662b94459f8f1a084e55637ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099c9c0b347544fdb0dcd7677fa7da29" is_pre_election: true
I20260619 17:00:36.770798 3882 raft_consensus.cc:2454] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d85b2662b94459f8f1a084e55637ce6 in term 0.
I20260619 17:00:36.770947 3748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd8ccaebfc142d2889b2005d4b5b40d" candidate_uuid: "1d85b2662b94459f8f1a084e55637ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130ad2350d644837a5f9c4cca58910df" is_pre_election: true
I20260619 17:00:36.771425 3748 raft_consensus.cc:2454] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d85b2662b94459f8f1a084e55637ce6 in term 0.
I20260619 17:00:36.771528 3547 leader_election.cc:302] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099c9c0b347544fdb0dcd7677fa7da29, 1d85b2662b94459f8f1a084e55637ce6; no voters:
I20260619 17:00:36.772022 3952 raft_consensus.cc:2781] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 17:00:36.772214 3952 raft_consensus.cc:482] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 17:00:36.772338 3952 raft_consensus.cc:3037] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:36.774608 3952 raft_consensus.cc:504] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } }
I20260619 17:00:36.775302 3952 leader_election.cc:288] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [CANDIDATE]: Term 1 election: Requested vote from peers 130ad2350d644837a5f9c4cca58910df (127.16.48.2:38871), 099c9c0b347544fdb0dcd7677fa7da29 (127.16.48.3:35819)
I20260619 17:00:36.775906 3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd8ccaebfc142d2889b2005d4b5b40d" candidate_uuid: "1d85b2662b94459f8f1a084e55637ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099c9c0b347544fdb0dcd7677fa7da29"
I20260619 17:00:36.775980 3748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd8ccaebfc142d2889b2005d4b5b40d" candidate_uuid: "1d85b2662b94459f8f1a084e55637ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130ad2350d644837a5f9c4cca58910df"
I20260619 17:00:36.776185 3882 raft_consensus.cc:3037] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:36.776286 3748 raft_consensus.cc:3037] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:36.778671 3882 raft_consensus.cc:2454] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d85b2662b94459f8f1a084e55637ce6 in term 1.
I20260619 17:00:36.778784 3748 raft_consensus.cc:2454] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d85b2662b94459f8f1a084e55637ce6 in term 1.
I20260619 17:00:36.779322 3547 leader_election.cc:302] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099c9c0b347544fdb0dcd7677fa7da29, 1d85b2662b94459f8f1a084e55637ce6; no voters:
I20260619 17:00:36.779675 3952 raft_consensus.cc:2781] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:36.780197 3952 raft_consensus.cc:686] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 1d85b2662b94459f8f1a084e55637ce6, State: Running, Role: LEADER
I20260619 17:00:36.780761 3952 consensus_queue.cc:237] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } }
I20260619 17:00:36.786069 3470 catalog_manager.cc:5495] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d85b2662b94459f8f1a084e55637ce6 (127.16.48.1). New cstate: current_term: 1 leader_uuid: "1d85b2662b94459f8f1a084e55637ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d85b2662b94459f8f1a084e55637ce6" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 35793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 } health_report { overall_health: UNKNOWN } } }
W20260619 17:00:36.819267 3794 tablet.cc:2369] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:36.839033 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.833638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.16.48.1:35793)
0619 17:00:36.833852 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:36.833856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:36.833874 (+ 18us) server_negotiation.cc:407] Connection header received
0619 17:00:36.833975 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:36.833979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:36.834078 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:36.834213 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:00:36.834546 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.835267 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.836784 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.837263 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.837394 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.837914 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:00:36.838489 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:00:36.838615 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:36.838706 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 17:00:36.838786 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 17:00:36.848522 3748 raft_consensus.cc:1261] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [term 1 FOLLOWER]: Refusing update from remote peer 1d85b2662b94459f8f1a084e55637ce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:36.848523 3882 raft_consensus.cc:1261] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [term 1 FOLLOWER]: Refusing update from remote peer 1d85b2662b94459f8f1a084e55637ce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:36.849705 3957 consensus_queue.cc:1035] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "130ad2350d644837a5f9c4cca58910df" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 38871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:36.850109 3952 consensus_queue.cc:1035] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "099c9c0b347544fdb0dcd7677fa7da29" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 35819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:36.864354 3961 mvcc.cc:205] Tried to move back new op lower bound from 7298615037319323648 to 7298615037058117632. Current Snapshot: MvccSnapshot[applied={T|T < 7298615037319323648}]
I20260619 17:00:36.879515 3469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:36.879827 3469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:36.883988 3469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486b4832935f4c19866fe8a3bd9dc57d: Sending DeleteTablet for 3 replicas of tablet afd8ccaebfc142d2889b2005d4b5b40d
I20260619 17:00:36.885557 3728 tablet_service.cc:1514] Processing DeleteTablet for tablet afd8ccaebfc142d2889b2005d4b5b40d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:36 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:58722
I20260619 17:00:36.885564 3594 tablet_service.cc:1514] Processing DeleteTablet for tablet afd8ccaebfc142d2889b2005d4b5b40d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:36 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:34194
I20260619 17:00:36.885557 3862 tablet_service.cc:1514] Processing DeleteTablet for tablet afd8ccaebfc142d2889b2005d4b5b40d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:36 UTC) from {username='kudu', principal='kudu/127.16.48.62@KRBTEST.COM'} at 127.0.0.1:38842
I20260619 17:00:36.892042 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 3531
I20260619 17:00:36.893893 3972 tablet_replica.cc:330] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: stopping tablet replica
I20260619 17:00:36.894536 3972 raft_consensus.cc:2229] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:36.895231 3972 raft_consensus.cc:2258] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:36.897825 3972 ts_tablet_manager.cc:1905] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:36.897866 3970 tablet_replica.cc:330] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: stopping tablet replica
I20260619 17:00:36.898406 3970 raft_consensus.cc:2229] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:36.899143 3970 raft_consensus.cc:2258] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:36.901926 3970 ts_tablet_manager.cc:1905] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:36.905998 3972 ts_tablet_manager.cc:1918] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:36.906298 3972 log.cc:1196] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/afd8ccaebfc142d2889b2005d4b5b40d
I20260619 17:00:36.907081 3972 ts_tablet_manager.cc:1939] T afd8ccaebfc142d2889b2005d4b5b40d P 099c9c0b347544fdb0dcd7677fa7da29: Deleting consensus metadata
I20260619 17:00:36.910113 3454 catalog_manager.cc:4841] TS 099c9c0b347544fdb0dcd7677fa7da29 (127.16.48.3:35819): tablet afd8ccaebfc142d2889b2005d4b5b40d (table test-table [id=79ecc354ddd24bbf948cf3bdd3eea787]) successfully deleted
I20260619 17:00:36.911221 3971 tablet_replica.cc:330] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: stopping tablet replica
I20260619 17:00:36.911330 3970 ts_tablet_manager.cc:1918] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:36.911631 3970 log.cc:1196] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/afd8ccaebfc142d2889b2005d4b5b40d
I20260619 17:00:36.911886 3971 raft_consensus.cc:2229] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 17:00:36.912412 3970 ts_tablet_manager.cc:1939] T afd8ccaebfc142d2889b2005d4b5b40d P 130ad2350d644837a5f9c4cca58910df: Deleting consensus metadata
I20260619 17:00:36.912753 3971 raft_consensus.cc:2258] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:36.915066 3456 catalog_manager.cc:4841] TS 130ad2350d644837a5f9c4cca58910df (127.16.48.2:38871): tablet afd8ccaebfc142d2889b2005d4b5b40d (table test-table [id=79ecc354ddd24bbf948cf3bdd3eea787]) successfully deleted
I20260619 17:00:36.915310 3971 ts_tablet_manager.cc:1905] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 17:00:36 dist-test-slave-3ffd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.1@KRBTEST.COM
I20260619 17:00:36.923661 3971 ts_tablet_manager.cc:1918] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:36.923878 3971 log.cc:1196] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/afd8ccaebfc142d2889b2005d4b5b40d
I20260619 17:00:36.924548 3971 ts_tablet_manager.cc:1939] T afd8ccaebfc142d2889b2005d4b5b40d P 1d85b2662b94459f8f1a084e55637ce6: Deleting consensus metadata
I20260619 17:00:36.926708 3456 catalog_manager.cc:4841] TS 1d85b2662b94459f8f1a084e55637ce6 (127.16.48.1:35793): tablet afd8ccaebfc142d2889b2005d4b5b40d (table test-table [id=79ecc354ddd24bbf948cf3bdd3eea787]) successfully deleted
I20260619 17:00:36.928844 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:36.893001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.16.48.1:35793)
0619 17:00:36.893223 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:36.893231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:36.910869 (+ 17638us) server_negotiation.cc:407] Connection header received
0619 17:00:36.911510 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:36.911529 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:36.911637 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:36.911733 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:36.913862 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.914839 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.916747 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:36.917282 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:36.917444 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:36.921812 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:36.921827 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:36.921844 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:36.921892 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:36.925594 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:36.926434 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:36.926447 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:36.926451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:36.926679 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:36.927144 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:36.927150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:36.927153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:36.927533 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:36.927866 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:36.928224 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 17:00:36.928496 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 17:00:36.929519 3654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:37.033530 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 3531
I20260619 17:00:37.055778 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 3665
Jun 19 17:00:37 dist-test-slave-3ffd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.2@KRBTEST.COM
I20260619 17:00:37.072180 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:37.056762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.16.48.2:38871)
0619 17:00:37.056971 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:37.056977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:37.056997 (+ 20us) server_negotiation.cc:407] Connection header received
0619 17:00:37.057170 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:37.057180 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:37.057265 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:37.057379 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:37.058707 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:37.059488 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:37.061076 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:37.061526 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:37.061648 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:37.066183 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:37.066201 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:37.066218 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:37.066263 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:37.069259 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:37.070011 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:37.070019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:37.070022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:37.070170 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:37.070507 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:37.070510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:37.070512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:37.070921 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:37.071163 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:37.071500 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 17:00:37.071764 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 17:00:37.072885 3788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:37.165025 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 3665
I20260619 17:00:37.183852 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 3799
Jun 19 17:00:37 dist-test-slave-3ffd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.3@KRBTEST.COM
I20260619 17:00:37.207091 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:37.184753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38844 (local address 127.16.48.3:35819)
0619 17:00:37.184958 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:37.184963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:37.184984 (+ 21us) server_negotiation.cc:407] Connection header received
0619 17:00:37.185149 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:37.185153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:37.185221 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:37.185340 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:37.186956 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:37.187692 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:37.189127 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:37.189515 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:37.189621 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:37.200789 (+ 11168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:37.200807 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:37.200825 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:37.200875 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:37.203777 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:37.204794 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:37.204804 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:37.204806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:37.204984 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:37.205402 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:37.205406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:37.205408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:37.205839 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:37.206084 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:37.206452 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 17:00:37.206661 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 17:00:37.207788 3922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:37.298496 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 3799
I20260619 17:00:37.315665 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 3433
I20260619 17:00:37.316969 3500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:37.430694 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 3433
2026-06-19T17:00:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T17:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:37Z Disabled control of system clock
I20260619 17:00:37.514673 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:33989
--webserver_interface=127.16.48.62
--webserver_port=0
--builtin_ntp_servers=127.16.48.20:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:33989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 17:00:37.874202 3981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:37.874533 3981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:37.874683 3981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:37.882551 3981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:37.882686 3981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:37.882772 3981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:37.882860 3981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:37.882973 3981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:37.893631 3981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:33989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:33989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:37.895087 3981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:37.897100 3981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:37.910221 3987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:37.910295 3986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:37.910318 3989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:37.911191 3981 server_base.cc:1034] running on GCE node
I20260619 17:00:37.912089 3981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:37.913794 3981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:37.915022 3981 hybrid_clock.cc:648] HybridClock initialized: now 1781888437914964 us; error 52 us; skew 500 ppm
I20260619 17:00:37.915427 3981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:37.917954 3981 webserver.cc:460] Webserver started at http://127.16.48.62:46411/ using document root <none> and password file <none>
I20260619 17:00:37.918741 3981 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:37.918841 3981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:37.919225 3981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:37.922573 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "4417b21fac4e4cb3a560025cfe26d046"
format_stamp: "Formatted at 2026-06-19 17:00:37 on dist-test-slave-3ffd"
server_key: "4605280615f4f6693b4f44d56e5e4445"
server_key_iv: "e867bfbfe7bc9b97cd74b1721fee6d01"
server_key_version: "encryptionkey@0"
I20260619 17:00:37.923471 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "4417b21fac4e4cb3a560025cfe26d046"
format_stamp: "Formatted at 2026-06-19 17:00:37 on dist-test-slave-3ffd"
server_key: "4605280615f4f6693b4f44d56e5e4445"
server_key_iv: "e867bfbfe7bc9b97cd74b1721fee6d01"
server_key_version: "encryptionkey@0"
I20260619 17:00:37.929335 3981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 17:00:37.933249 3995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:37.934755 3981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 17:00:37.934983 3981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "4417b21fac4e4cb3a560025cfe26d046"
format_stamp: "Formatted at 2026-06-19 17:00:37 on dist-test-slave-3ffd"
server_key: "4605280615f4f6693b4f44d56e5e4445"
server_key_iv: "e867bfbfe7bc9b97cd74b1721fee6d01"
server_key_version: "encryptionkey@0"
I20260619 17:00:37.935262 3981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:37.950305 3981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:37.951314 3981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:37.951655 3981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:37.972082 3981 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:33989
I20260619 17:00:37.972110 4046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:33989 every 8 connection(s)
I20260619 17:00:37.973596 3981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:37.977770 4047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:37.982223 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 3981
I20260619 17:00:37.982486 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:37.983006 16576 external_mini_cluster.cc:1428] Setting key 6c2f022c3fdedc4311656eff44746e6f
I20260619 17:00:37.991889 4047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046: Bootstrap starting.
I20260619 17:00:37.995959 4047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:37.997506 4047 log.cc:824] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:38.000716 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:37.985187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.16.48.62:33989)
0619 17:00:37.986700 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:37.986722 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:37.986789 (+ 67us) server_negotiation.cc:407] Connection header received
0619 17:00:37.989795 (+ 3006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:37.989834 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:37.990563 (+ 729us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:37.991072 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:37.991683 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:37.993729 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:37.995169 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:37.996349 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:37.996595 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:37.997877 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:37.997889 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:37.997908 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:37.998673 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:37.998771 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:37.998794 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:37.998941 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:37.999220 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 17:00:37.999585 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":675,"threads_started":1}
I20260619 17:00:38.001109 4047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046: No bootstrap required, opened a new log
I20260619 17:00:38.009696 4047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4417b21fac4e4cb3a560025cfe26d046" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33989 } }
I20260619 17:00:38.010133 4047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:38.010215 4047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4417b21fac4e4cb3a560025cfe26d046, State: Initialized, Role: FOLLOWER
I20260619 17:00:38.010793 4047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4417b21fac4e4cb3a560025cfe26d046" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33989 } }
I20260619 17:00:38.011022 4047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:38.011130 4047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:38.011271 4047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:38.013703 4047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4417b21fac4e4cb3a560025cfe26d046" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33989 } }
I20260619 17:00:38.014233 4047 leader_election.cc:302] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4417b21fac4e4cb3a560025cfe26d046; no voters:
I20260619 17:00:38.015009 4047 leader_election.cc:288] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:38.015300 4052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:38.016105 4052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [term 1 LEADER]: Becoming Leader. State: Replica: 4417b21fac4e4cb3a560025cfe26d046, State: Running, Role: LEADER
I20260619 17:00:38.016829 4052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4417b21fac4e4cb3a560025cfe26d046" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33989 } }
I20260619 17:00:38.017488 4047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:38.020356 4054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4417b21fac4e4cb3a560025cfe26d046. Latest consensus state: current_term: 1 leader_uuid: "4417b21fac4e4cb3a560025cfe26d046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4417b21fac4e4cb3a560025cfe26d046" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33989 } } }
I20260619 17:00:38.020354 4053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4417b21fac4e4cb3a560025cfe26d046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4417b21fac4e4cb3a560025cfe26d046" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33989 } } }
I20260619 17:00:38.020967 4054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:38.020988 4053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:38.025985 4058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:38.034175 4058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:38.046420 4058 catalog_manager.cc:1347] Generated new cluster ID: 32b01f64a6da442ca50847ef9cef93a4
I20260619 17:00:38.046608 4058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:38.075812 4058 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:38.077917 4058 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:38.088304 4058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046: Generated new TSK 0
I20260619 17:00:38.089556 4058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:38.104516 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.1:0
--local_ip_for_outbound_sockets=127.16.48.1
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:33989
--builtin_ntp_servers=127.16.48.20:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 17:00:38.484263 4071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:38.484632 4071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:38.484797 4071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:38.493242 4071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:38.493410 4071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:38.493593 4071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.1
I20260619 17:00:38.503230 4071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.48.1
--webserver_port=0
--tserver_master_addrs=127.16.48.62:33989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.1
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:38.504722 4071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:38.506666 4071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:38.521225 4079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:38.521760 4077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:38.522018 4076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:38.522296 4071 server_base.cc:1034] running on GCE node
I20260619 17:00:38.523007 4071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:38.524048 4071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:38.525326 4071 hybrid_clock.cc:648] HybridClock initialized: now 1781888438525270 us; error 65 us; skew 500 ppm
I20260619 17:00:38.525672 4071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:38.528170 4071 webserver.cc:460] Webserver started at http://127.16.48.1:40861/ using document root <none> and password file <none>
I20260619 17:00:38.528931 4071 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:38.529081 4071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:38.529429 4071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:38.532685 4071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/data/instance:
uuid: "0e3dde2745c24c49b1cbf92350b2cf4e"
format_stamp: "Formatted at 2026-06-19 17:00:38 on dist-test-slave-3ffd"
server_key: "0d594aa4d3f04ad6d74ed58f123088c4"
server_key_iv: "2c39d46617cd023a4e92db76e4faaf4b"
server_key_version: "encryptionkey@0"
I20260619 17:00:38.533649 4071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance:
uuid: "0e3dde2745c24c49b1cbf92350b2cf4e"
format_stamp: "Formatted at 2026-06-19 17:00:38 on dist-test-slave-3ffd"
server_key: "0d594aa4d3f04ad6d74ed58f123088c4"
server_key_iv: "2c39d46617cd023a4e92db76e4faaf4b"
server_key_version: "encryptionkey@0"
I20260619 17:00:38.539763 4071 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 17:00:38.543857 4085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:38.545471 4071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 17:00:38.545713 4071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/wal
uuid: "0e3dde2745c24c49b1cbf92350b2cf4e"
format_stamp: "Formatted at 2026-06-19 17:00:38 on dist-test-slave-3ffd"
server_key: "0d594aa4d3f04ad6d74ed58f123088c4"
server_key_iv: "2c39d46617cd023a4e92db76e4faaf4b"
server_key_version: "encryptionkey@0"
I20260619 17:00:38.545976 4071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:38.565099 4071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:38.566092 4071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:38.566416 4071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:38.567565 4071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:38.569300 4071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:38.569574 4071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:38.569738 4071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:38.569839 4071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:38.605739 4071 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.1:40975
I20260619 17:00:38.605796 4197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.1:40975 every 8 connection(s)
I20260619 17:00:38.607241 4071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/data/info.pb
I20260619 17:00:38.616873 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 4071
I20260619 17:00:38.617089 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/wal/instance
I20260619 17:00:38.617532 16576 external_mini_cluster.cc:1428] Setting key 2773608ef9da60fcfd64ffa5381aa2ee
I20260619 17:00:38.620824 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.2:0
--local_ip_for_outbound_sockets=127.16.48.2
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:33989
--builtin_ntp_servers=127.16.48.20:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 17:00:38.621940 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:38.610123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:56563 (local address 127.16.48.62:33989)
0619 17:00:38.610780 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:38.610790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:38.612274 (+ 1484us) server_negotiation.cc:407] Connection header received
0619 17:00:38.614218 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:38.614224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:38.614328 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:38.614454 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:38.615641 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:38.616456 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:38.618443 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:38.619073 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:38.619209 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:38.621113 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:38.621121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:38.621128 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:38.621153 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:38.621196 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:38.621209 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:38.621282 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:38.621581 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 17:00:38.621667 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":304,"threads_started":1}
I20260619 17:00:38.623620 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:38.610930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:33989 (local address 127.16.48.1:56563)
0619 17:00:38.612027 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:38.612082 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:38.613885 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:38.614721 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:38.614730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:38.614829 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:38.615447 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:38.615465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:38.616629 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:38.616633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:38.618282 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:38.618299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:38.619413 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:38.619423 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:38.619764 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:38.620620 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 17:00:38.620627 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:38.620691 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 17:00:38.620820 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:38.621316 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:38.621331 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:38.621342 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 17:00:38.621566 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 17:00:38.622066 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":354,"threads_started":1}
I20260619 17:00:38.625434 4198 heartbeater.cc:347] Connected to a master server at 127.16.48.62:33989
I20260619 17:00:38.625785 4198 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:38.626610 4198 heartbeater.cc:511] Master 127.16.48.62:33989 requested a full tablet report, sending...
I20260619 17:00:38.628981 4012 ts_manager.cc:194] Registered new tserver with Master: 0e3dde2745c24c49b1cbf92350b2cf4e (127.16.48.1:40975)
I20260619 17:00:38.631129 4012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.1:56563
W20260619 17:00:38.981521 4202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:38.981940 4202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:38.982074 4202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:38.994731 4202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:38.994899 4202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:38.995082 4202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.2
I20260619 17:00:39.006335 4202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.48.2
--webserver_port=0
--tserver_master_addrs=127.16.48.62:33989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.2
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:39.007794 4202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:39.009872 4202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:39.024516 4207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:39.024500 4210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:39.024519 4208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:39.025629 4202 server_base.cc:1034] running on GCE node
I20260619 17:00:39.026304 4202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:39.027402 4202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:39.028618 4202 hybrid_clock.cc:648] HybridClock initialized: now 1781888439028547 us; error 65 us; skew 500 ppm
I20260619 17:00:39.028988 4202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:39.031629 4202 webserver.cc:460] Webserver started at http://127.16.48.2:36993/ using document root <none> and password file <none>
I20260619 17:00:39.032493 4202 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:39.032629 4202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:39.032995 4202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:39.036252 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/data/instance:
uuid: "708919e6fbfa425f8a8988fddbf1f0ff"
format_stamp: "Formatted at 2026-06-19 17:00:39 on dist-test-slave-3ffd"
server_key: "d6f339db5b517c6279afa59d96ced509"
server_key_iv: "b9e21700df06ec062d0c975701b79cd2"
server_key_version: "encryptionkey@0"
I20260619 17:00:39.037194 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance:
uuid: "708919e6fbfa425f8a8988fddbf1f0ff"
format_stamp: "Formatted at 2026-06-19 17:00:39 on dist-test-slave-3ffd"
server_key: "d6f339db5b517c6279afa59d96ced509"
server_key_iv: "b9e21700df06ec062d0c975701b79cd2"
server_key_version: "encryptionkey@0"
I20260619 17:00:39.043372 4202 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 17:00:39.047655 4216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:39.049233 4202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 17:00:39.049460 4202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/wal
uuid: "708919e6fbfa425f8a8988fddbf1f0ff"
format_stamp: "Formatted at 2026-06-19 17:00:39 on dist-test-slave-3ffd"
server_key: "d6f339db5b517c6279afa59d96ced509"
server_key_iv: "b9e21700df06ec062d0c975701b79cd2"
server_key_version: "encryptionkey@0"
I20260619 17:00:39.049727 4202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:39.064378 4202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:39.065452 4202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:39.065802 4202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:39.066965 4202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:39.068807 4202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:39.068949 4202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:39.069103 4202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:39.069193 4202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:39.104765 4202 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.2:40813
I20260619 17:00:39.104822 4328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.2:40813 every 8 connection(s)
I20260619 17:00:39.106314 4202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/data/info.pb
I20260619 17:00:39.112059 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 4202
I20260619 17:00:39.112334 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/wal/instance
I20260619 17:00:39.112766 16576 external_mini_cluster.cc:1428] Setting key fcd913f1717b564853858fb7bce4ff23
I20260619 17:00:39.116042 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.48.3:0
--local_ip_for_outbound_sockets=127.16.48.3
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:33989
--builtin_ntp_servers=127.16.48.20:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 17:00:39.120436 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.109087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:53981 (local address 127.16.48.62:33989)
0619 17:00:39.109309 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:39.109316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:39.110942 (+ 1626us) server_negotiation.cc:407] Connection header received
0619 17:00:39.113082 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:39.113089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:39.113194 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:39.113325 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:39.114614 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.115495 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.117261 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.117804 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.117941 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.119533 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:39.119538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:39.119541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:39.119562 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:39.119613 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:39.119626 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:39.119728 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:39.119965 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 17:00:39.120073 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 17:00:39.121628 4331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.109770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:33989 (local address 127.16.48.2:53981)
0619 17:00:39.110764 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:39.110809 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:39.112717 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:39.113630 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:39.113646 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:39.113784 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:39.114409 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.114428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.115690 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.115694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.117083 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.117108 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.118058 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.118068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.118319 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.119146 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 17:00:39.119149 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:39.119194 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 17:00:39.119320 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:39.119730 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:39.119746 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:39.119755 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 17:00:39.119954 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 17:00:39.120447 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":325,"threads_started":1}
I20260619 17:00:39.123334 4329 heartbeater.cc:347] Connected to a master server at 127.16.48.62:33989
I20260619 17:00:39.123678 4329 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:39.124795 4329 heartbeater.cc:511] Master 127.16.48.62:33989 requested a full tablet report, sending...
I20260619 17:00:39.126757 4012 ts_manager.cc:194] Registered new tserver with Master: 708919e6fbfa425f8a8988fddbf1f0ff (127.16.48.2:40813)
I20260619 17:00:39.128091 4012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.2:53981
W20260619 17:00:39.478986 4332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:39.479317 4332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:39.479417 4332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:39.487195 4332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:39.487329 4332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:39.487449 4332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.48.3
I20260619 17:00:39.497677 4332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.48.3
--webserver_port=0
--tserver_master_addrs=127.16.48.62:33989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.48.3
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:39.499032 4332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:39.500998 4332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:39.515334 4340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:39.515269 4337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:39.515367 4338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:39.516574 4332 server_base.cc:1034] running on GCE node
I20260619 17:00:39.517313 4332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:39.518422 4332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:39.519651 4332 hybrid_clock.cc:648] HybridClock initialized: now 1781888439519584 us; error 58 us; skew 500 ppm
I20260619 17:00:39.520017 4332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 17:00:39.522507 4332 webserver.cc:460] Webserver started at http://127.16.48.3:34419/ using document root <none> and password file <none>
I20260619 17:00:39.523277 4332 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:39.523401 4332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:39.523757 4332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:39.527240 4332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/data/instance:
uuid: "a9bcb898ce2a4705810406ca8081fcd5"
format_stamp: "Formatted at 2026-06-19 17:00:39 on dist-test-slave-3ffd"
server_key: "6c65211a1405c06ea34cd04fbf937ffa"
server_key_iv: "37f94bcfec46ffd82f72ff0f4a882190"
server_key_version: "encryptionkey@0"
I20260619 17:00:39.528311 4332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance:
uuid: "a9bcb898ce2a4705810406ca8081fcd5"
format_stamp: "Formatted at 2026-06-19 17:00:39 on dist-test-slave-3ffd"
server_key: "6c65211a1405c06ea34cd04fbf937ffa"
server_key_iv: "37f94bcfec46ffd82f72ff0f4a882190"
server_key_version: "encryptionkey@0"
I20260619 17:00:39.534343 4332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 17:00:39.538776 4346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:39.540598 4332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 17:00:39.540822 4332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/wal
uuid: "a9bcb898ce2a4705810406ca8081fcd5"
format_stamp: "Formatted at 2026-06-19 17:00:39 on dist-test-slave-3ffd"
server_key: "6c65211a1405c06ea34cd04fbf937ffa"
server_key_iv: "37f94bcfec46ffd82f72ff0f4a882190"
server_key_version: "encryptionkey@0"
I20260619 17:00:39.541079 4332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:39.566979 4332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:39.567914 4332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:39.568297 4332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:39.569399 4332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 17:00:39.571081 4332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 17:00:39.571218 4332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:39.571363 4332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 17:00:39.571463 4332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:39.607280 4332 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.3:45217
I20260619 17:00:39.607328 4458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.3:45217 every 8 connection(s)
I20260619 17:00:39.608830 4332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/data/info.pb
I20260619 17:00:39.616786 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 4332
I20260619 17:00:39.616966 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/wal/instance
I20260619 17:00:39.617630 16576 external_mini_cluster.cc:1428] Setting key 464f0b303e2fea448966fa6595b955d0
I20260619 17:00:39.623648 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.611746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.3:41273 (local address 127.16.48.62:33989)
0619 17:00:39.611974 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:39.611981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:39.613659 (+ 1678us) server_negotiation.cc:407] Connection header received
0619 17:00:39.615629 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:39.615634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:39.615715 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:39.615816 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:39.617235 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.618428 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.620412 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.620986 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.621143 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.622724 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:39.622729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:39.622732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:39.622758 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:39.622792 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:39.622804 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:39.622901 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:39.623173 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 17:00:39.623284 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 17:00:39.624883 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.612433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:33989 (local address 127.16.48.3:41273)
0619 17:00:39.613462 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:39.613508 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:39.615259 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:39.616058 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:39.616067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:39.616253 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:00:39.616935 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.616968 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.618634 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.618639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.620249 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.620266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.621258 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.621267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.621531 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.622329 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 17:00:39.622331 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:00:39.622378 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 17:00:39.622480 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:00:39.622915 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:00:39.622930 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:00:39.622953 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 17:00:39.623144 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 17:00:39.623614 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":316,"threads_started":1}
I20260619 17:00:39.626502 4459 heartbeater.cc:347] Connected to a master server at 127.16.48.62:33989
I20260619 17:00:39.626884 4459 heartbeater.cc:464] Registering TS with master...
I20260619 17:00:39.627719 4459 heartbeater.cc:511] Master 127.16.48.62:33989 requested a full tablet report, sending...
I20260619 17:00:39.629375 4012 ts_manager.cc:194] Registered new tserver with Master: a9bcb898ce2a4705810406ca8081fcd5 (127.16.48.3:45217)
I20260619 17:00:39.630249 4012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.48.3:41273
I20260619 17:00:39.633421 16576 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 17:00:39.635385 4198 heartbeater.cc:503] Master 127.16.48.62:33989 was elected leader, sending a full tablet report...
I20260619 17:00:39.644340 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.638294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.16.48.62:33989)
0619 17:00:39.638538 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:39.638547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:39.638938 (+ 391us) server_negotiation.cc:407] Connection header received
0619 17:00:39.639066 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:39.639070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:39.639135 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:39.639222 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:39.639743 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.640620 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.641964 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.642374 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.642506 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.643536 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:39.643541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:39.643544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:39.643565 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:39.643599 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:39.643612 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:39.643688 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:39.643847 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 17:00:39.644002 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 17:00:39.648753 4012 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 17:00:39.656067 4012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 17:00:39.681923 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.672405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.16.48.1:40975)
0619 17:00:39.673166 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:39.673176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:39.673210 (+ 34us) server_negotiation.cc:407] Connection header received
0619 17:00:39.673331 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:39.673340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:39.673614 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:39.673845 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:39.675361 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.676945 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.679249 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.680610 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.680786 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.681118 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:39.681349 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 17:00:39.681523 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":383,"threads_started":1}
I20260619 17:00:39.682005 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.671791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:40975 (local address 127.0.0.1:46444)
0619 17:00:39.672724 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:39.672747 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:39.672864 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:39.673854 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:39.673865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:39.673895 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:39.675136 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.675151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.677081 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.677089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.678965 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.678982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.680819 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.680826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.681052 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.681121 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 17:00:39.681292 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 17:00:39.681508 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":32,"spinlock_wait_cycles":17280,"thread_start_us":620,"threads_started":1}
I20260619 17:00:39.682808 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.669580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:40813 (local address 127.0.0.1:40720)
0619 17:00:39.672003 (+ 2423us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:39.672034 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:39.672194 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:39.674034 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:39.674037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:39.674098 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:39.674796 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.674809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.676892 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.676899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.679528 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.679545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.681629 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.681634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.681909 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.682043 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 17:00:39.682125 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 17:00:39.682361 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2277,"mutex_wait_us":269,"spinlock_wait_cycles":44160,"thread_start_us":349,"threads_started":1}
I20260619 17:00:39.683054 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.669497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:45217 (local address 127.0.0.1:32890)
0619 17:00:39.670416 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:39.670462 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:39.670657 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:39.674026 (+ 3369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:39.674031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:39.674066 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:39.674773 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.674787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.676632 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.676639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.679536 (+ 2897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:39.679545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.681625 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.681632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:39.681910 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.682045 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 17:00:39.682125 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 17:00:39.682361 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":368,"spinlock_wait_cycles":2816,"thread_start_us":2186,"threads_started":1}
I20260619 17:00:39.683733 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.669958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.16.48.3:45217)
0619 17:00:39.673503 (+ 3545us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:39.673512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:39.673552 (+ 40us) server_negotiation.cc:407] Connection header received
0619 17:00:39.673640 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:39.673646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:39.673814 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:39.674472 (+ 658us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:39.674980 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.676486 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.679805 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.681361 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.682741 (+ 1380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.683090 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:39.683174 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 17:00:39.683365 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3375,"thread_start_us":3327,"threads_started":1}
I20260619 17:00:39.684010 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:39.670550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.16.48.2:40813)
0619 17:00:39.673398 (+ 2848us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:39.673409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:39.673437 (+ 28us) server_negotiation.cc:407] Connection header received
0619 17:00:39.673548 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:39.673555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:39.673788 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:39.674204 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:39.675006 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.676745 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.679813 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:39.681361 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:39.683144 (+ 1783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:39.683382 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:39.683459 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 17:00:39.683641 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2684,"thread_start_us":2531,"threads_started":1}
I20260619 17:00:39.687080 4394 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6233a6359f4df5a3b1922bd698633e (DEFAULT_TABLE table=test-table [id=9f45428438924a48a76e33db680781dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:39.687148 4133 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6233a6359f4df5a3b1922bd698633e (DEFAULT_TABLE table=test-table [id=9f45428438924a48a76e33db680781dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:39.687592 4264 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6233a6359f4df5a3b1922bd698633e (DEFAULT_TABLE table=test-table [id=9f45428438924a48a76e33db680781dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 17:00:39.688642 4394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6233a6359f4df5a3b1922bd698633e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:39.688930 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6233a6359f4df5a3b1922bd698633e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:39.688941 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6233a6359f4df5a3b1922bd698633e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:39.703225 4477 tablet_bootstrap.cc:492] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: Bootstrap starting.
I20260619 17:00:39.707135 4479 tablet_bootstrap.cc:492] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: Bootstrap starting.
I20260619 17:00:39.709043 4478 tablet_bootstrap.cc:492] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: Bootstrap starting.
I20260619 17:00:39.709257 4477 tablet_bootstrap.cc:654] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:39.710567 4477 log.cc:824] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:39.711351 4479 tablet_bootstrap.cc:654] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:39.712762 4479 log.cc:824] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:39.712986 4478 tablet_bootstrap.cc:654] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:39.714197 4477 tablet_bootstrap.cc:492] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: No bootstrap required, opened a new log
I20260619 17:00:39.714574 4478 log.cc:824] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:39.714605 4477 ts_tablet_manager.cc:1397] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260619 17:00:39.717133 4479 tablet_bootstrap.cc:492] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: No bootstrap required, opened a new log
I20260619 17:00:39.717545 4479 ts_tablet_manager.cc:1397] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260619 17:00:39.717823 4478 tablet_bootstrap.cc:492] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: No bootstrap required, opened a new log
I20260619 17:00:39.718112 4478 ts_tablet_manager.cc:1397] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 17:00:39.723079 4477 raft_consensus.cc:348] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:39.723533 4477 raft_consensus.cc:374] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:39.723655 4477 raft_consensus.cc:729] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bcb898ce2a4705810406ca8081fcd5, State: Initialized, Role: FOLLOWER
I20260619 17:00:39.724617 4477 consensus_queue.cc:260] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:39.726228 4459 heartbeater.cc:503] Master 127.16.48.62:33989 was elected leader, sending a full tablet report...
I20260619 17:00:39.727056 4477 ts_tablet_manager.cc:1428] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 17:00:39.728828 4478 raft_consensus.cc:348] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:39.729297 4478 raft_consensus.cc:374] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:39.729436 4478 raft_consensus.cc:729] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708919e6fbfa425f8a8988fddbf1f0ff, State: Initialized, Role: FOLLOWER
I20260619 17:00:39.730156 4478 consensus_queue.cc:260] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:39.730144 4479 raft_consensus.cc:348] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:39.730671 4479 raft_consensus.cc:374] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:39.730790 4479 raft_consensus.cc:729] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3dde2745c24c49b1cbf92350b2cf4e, State: Initialized, Role: FOLLOWER
I20260619 17:00:39.731537 4479 consensus_queue.cc:260] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:39.731734 4329 heartbeater.cc:503] Master 127.16.48.62:33989 was elected leader, sending a full tablet report...
I20260619 17:00:39.732831 4478 ts_tablet_manager.cc:1428] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260619 17:00:39.734911 4479 ts_tablet_manager.cc:1428] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
W20260619 17:00:39.858691 4330 tablet.cc:2369] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 17:00:39.859864 4199 tablet.cc:2369] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 17:00:39.860694 4460 tablet.cc:2369] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:00:40.101755 4484 raft_consensus.cc:482] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 17:00:40.102133 4484 raft_consensus.cc:504] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:40.104048 4485 raft_consensus.cc:482] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 17:00:40.104471 4485 raft_consensus.cc:504] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:40.113626 4485 leader_election.cc:288] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9bcb898ce2a4705810406ca8081fcd5 (127.16.48.3:45217), 708919e6fbfa425f8a8988fddbf1f0ff (127.16.48.2:40813)
I20260619 17:00:40.104194 4484 leader_election.cc:288] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9bcb898ce2a4705810406ca8081fcd5 (127.16.48.3:45217), 0e3dde2745c24c49b1cbf92350b2cf4e (127.16.48.1:40975)
I20260619 17:00:40.128741 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.112078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:40335 (local address 127.16.48.1:40975)
0619 17:00:40.113252 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:40.113262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:40.118253 (+ 4991us) server_negotiation.cc:407] Connection header received
0619 17:00:40.119744 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:40.119750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:40.119864 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:40.119932 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:40.120855 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.122032 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.124985 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.126358 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.126462 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.126580 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:40.128284 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0619 17:00:40.128451 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967}
I20260619 17:00:40.128729 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.117951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:46987 (local address 127.16.48.2:40813)
0619 17:00:40.119420 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:40.119429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:40.119852 (+ 423us) server_negotiation.cc:407] Connection header received
0619 17:00:40.120343 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:40.120347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:40.120434 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:40.120529 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:40.121446 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.122669 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.125874 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.127439 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.127577 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.127676 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:40.128234 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 17:00:40.128385 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1303}
I20260619 17:00:40.129267 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.105264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.2:54755 (local address 127.16.48.3:45217)
0619 17:00:40.105542 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:40.105552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:40.118712 (+ 13160us) server_negotiation.cc:407] Connection header received
0619 17:00:40.119014 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:40.119021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:40.119102 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:40.119166 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:40.119970 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.121439 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.126728 (+ 5289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.128026 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.128207 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.128315 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:40.128852 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 17:00:40.129021 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 17:00:40.129650 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.108280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.2:40813 (local address 127.16.48.1:46987)
0619 17:00:40.119756 (+ 11476us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:40.119780 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:40.120181 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:40.120731 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:40.120738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:40.120770 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:40.121277 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:40.121289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.122868 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.122874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:40.125267 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:40.125284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.127814 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.127822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:40.128009 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.128045 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 17:00:40.129021 (+ 976us) client_negotiation.cc:240] Negotiation successful
0619 17:00:40.129180 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11327,"mutex_wait_us":6,"thread_start_us":11562,"threads_started":1}
I20260619 17:00:40.129767 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.104655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.1:40975 (local address 127.16.48.2:40335)
0619 17:00:40.118077 (+ 13422us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:40.118108 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:40.119498 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:40.120165 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:40.120171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:40.120196 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:40.120661 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:40.120672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.122221 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.122227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:40.124555 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:40.124573 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.127820 (+ 3247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.127833 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:40.128023 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.128073 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 17:00:40.129249 (+ 1176us) client_negotiation.cc:240] Negotiation successful
0619 17:00:40.129403 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13235,"mutex_wait_us":4,"thread_start_us":12558,"threads_started":1}
I20260619 17:00:40.130362 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.104593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:45217 (local address 127.16.48.2:54755)
0619 17:00:40.118553 (+ 13960us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:40.118581 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:40.118810 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:40.119360 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:40.119366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:40.119393 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:40.119804 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:40.119815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.121681 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.121686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:40.124530 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:40.124566 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.128450 (+ 3884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.128457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:40.128643 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.128673 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 17:00:40.129628 (+ 955us) client_negotiation.cc:240] Negotiation successful
0619 17:00:40.129811 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12691,"mutex_wait_us":6,"thread_start_us":12948,"threads_started":1}
I20260619 17:00:40.131127 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6233a6359f4df5a3b1922bd698633e" candidate_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" is_pre_election: true
I20260619 17:00:40.131620 4284 raft_consensus.cc:2454] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e3dde2745c24c49b1cbf92350b2cf4e in term 0.
I20260619 17:00:40.131608 4414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6233a6359f4df5a3b1922bd698633e" candidate_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bcb898ce2a4705810406ca8081fcd5" is_pre_election: true
I20260619 17:00:40.131556 4153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6233a6359f4df5a3b1922bd698633e" candidate_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" is_pre_election: true
I20260619 17:00:40.132063 4153 raft_consensus.cc:2454] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 708919e6fbfa425f8a8988fddbf1f0ff in term 0.
I20260619 17:00:40.132155 4414 raft_consensus.cc:2454] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 708919e6fbfa425f8a8988fddbf1f0ff in term 0.
I20260619 17:00:40.132834 4218 leader_election.cc:302] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3dde2745c24c49b1cbf92350b2cf4e, 708919e6fbfa425f8a8988fddbf1f0ff; no voters:
I20260619 17:00:40.133271 4484 raft_consensus.cc:2781] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 17:00:40.133262 4089 leader_election.cc:302] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3dde2745c24c49b1cbf92350b2cf4e, 708919e6fbfa425f8a8988fddbf1f0ff; no voters:
I20260619 17:00:40.133416 4484 raft_consensus.cc:482] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 17:00:40.133517 4484 raft_consensus.cc:3037] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:40.133701 4485 raft_consensus.cc:2781] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 17:00:40.133888 4485 raft_consensus.cc:482] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 17:00:40.134013 4485 raft_consensus.cc:3037] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:40.137228 4484 raft_consensus.cc:504] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:40.137480 4485 raft_consensus.cc:504] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:40.138586 4484 leader_election.cc:288] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [CANDIDATE]: Term 1 election: Requested vote from peers a9bcb898ce2a4705810406ca8081fcd5 (127.16.48.3:45217), 0e3dde2745c24c49b1cbf92350b2cf4e (127.16.48.1:40975)
I20260619 17:00:40.138655 4414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6233a6359f4df5a3b1922bd698633e" candidate_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bcb898ce2a4705810406ca8081fcd5"
I20260619 17:00:40.138890 4414 raft_consensus.cc:3037] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:40.139199 4153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6233a6359f4df5a3b1922bd698633e" candidate_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e"
I20260619 17:00:40.139513 4153 raft_consensus.cc:2379] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 708919e6fbfa425f8a8988fddbf1f0ff in current term 1: Already voted for candidate 0e3dde2745c24c49b1cbf92350b2cf4e in this term.
I20260619 17:00:40.139616 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.113719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.48.1:56781 (local address 127.16.48.3:45217)
0619 17:00:40.114384 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:40.114391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:40.127704 (+ 13313us) server_negotiation.cc:407] Connection header received
0619 17:00:40.127970 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:40.127982 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:40.128063 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:40.128143 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 17:00:40.132186 (+ 4043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.133489 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.136658 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.138021 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.138161 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.138280 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:40.138993 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 17:00:40.139182 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":301,"threads_started":1}
I20260619 17:00:40.140413 4485 leader_election.cc:288] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [CANDIDATE]: Term 1 election: Requested vote from peers a9bcb898ce2a4705810406ca8081fcd5 (127.16.48.3:45217), 708919e6fbfa425f8a8988fddbf1f0ff (127.16.48.2:40813)
I20260619 17:00:40.140493 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.112645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.3:45217 (local address 127.16.48.1:56781)
0619 17:00:40.127572 (+ 14927us) negotiation.cc:107] Waiting for socket to connect
0619 17:00:40.127591 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 17:00:40.127785 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:00:40.128439 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:00:40.128443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:00:40.128473 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 17:00:40.130428 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:40.130444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.133760 (+ 3316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.133768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:40.136362 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:00:40.136384 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.138489 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.138501 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:00:40.138785 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.138824 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 17:00:40.139945 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0619 17:00:40.140155 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14758,"thread_start_us":14492,"threads_started":1}
I20260619 17:00:40.141139 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6233a6359f4df5a3b1922bd698633e" candidate_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "708919e6fbfa425f8a8988fddbf1f0ff"
I20260619 17:00:40.141470 4284 raft_consensus.cc:2379] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0e3dde2745c24c49b1cbf92350b2cf4e in current term 1: Already voted for candidate 708919e6fbfa425f8a8988fddbf1f0ff in this term.
I20260619 17:00:40.141502 4413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6233a6359f4df5a3b1922bd698633e" candidate_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bcb898ce2a4705810406ca8081fcd5" is_pre_election: true
I20260619 17:00:40.141836 4412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6233a6359f4df5a3b1922bd698633e" candidate_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bcb898ce2a4705810406ca8081fcd5"
I20260619 17:00:40.142916 4086 leader_election.cc:302] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e3dde2745c24c49b1cbf92350b2cf4e; no voters: 708919e6fbfa425f8a8988fddbf1f0ff, a9bcb898ce2a4705810406ca8081fcd5
I20260619 17:00:40.143147 4414 raft_consensus.cc:2454] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708919e6fbfa425f8a8988fddbf1f0ff in term 1.
I20260619 17:00:40.143294 4485 raft_consensus.cc:2726] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 17:00:40.143596 4217 leader_election.cc:302] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 708919e6fbfa425f8a8988fddbf1f0ff, a9bcb898ce2a4705810406ca8081fcd5; no voters: 0e3dde2745c24c49b1cbf92350b2cf4e
I20260619 17:00:40.143970 4484 raft_consensus.cc:2781] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:40.144518 4484 raft_consensus.cc:686] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 1 LEADER]: Becoming Leader. State: Replica: 708919e6fbfa425f8a8988fddbf1f0ff, State: Running, Role: LEADER
I20260619 17:00:40.145092 4484 consensus_queue.cc:237] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } }
I20260619 17:00:40.150430 4011 catalog_manager.cc:5495] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 708919e6fbfa425f8a8988fddbf1f0ff (127.16.48.2). New cstate: current_term: 1 leader_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708919e6fbfa425f8a8988fddbf1f0ff" member_type: VOTER last_known_addr { host: "127.16.48.2" port: 40813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 } health_report { overall_health: UNKNOWN } } }
I20260619 17:00:40.253636 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.247390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.16.48.2:40813)
0619 17:00:40.247662 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:40.247669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:40.247994 (+ 325us) server_negotiation.cc:407] Connection header received
0619 17:00:40.248210 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:40.248215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:40.248308 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:40.248439 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:00:40.248904 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.249716 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.251283 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.251703 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.251804 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.252385 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:00:40.252939 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:00:40.253065 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:40.253190 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 17:00:40.253262 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 17:00:40.261934 4414 raft_consensus.cc:1261] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [term 1 FOLLOWER]: Refusing update from remote peer 708919e6fbfa425f8a8988fddbf1f0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:40.261974 4153 raft_consensus.cc:1261] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 1 FOLLOWER]: Refusing update from remote peer 708919e6fbfa425f8a8988fddbf1f0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:00:40.262887 4491 consensus_queue.cc:1035] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bcb898ce2a4705810406ca8081fcd5" member_type: VOTER last_known_addr { host: "127.16.48.3" port: 45217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:40.263269 4484 consensus_queue.cc:1035] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e3dde2745c24c49b1cbf92350b2cf4e" member_type: VOTER last_known_addr { host: "127.16.48.1" port: 40975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:00:40.298005 4012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:40.298353 4012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 17:00:40.302124 4012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4417b21fac4e4cb3a560025cfe26d046: Sending DeleteTablet for 3 replicas of tablet 2b6233a6359f4df5a3b1922bd698633e
I20260619 17:00:40.303486 4264 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6233a6359f4df5a3b1922bd698633e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:40 UTC) from {username='slave'} at 127.0.0.1:40720
I20260619 17:00:40.303761 4133 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6233a6359f4df5a3b1922bd698633e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:40 UTC) from {username='slave'} at 127.0.0.1:46444
I20260619 17:00:40.303769 4394 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6233a6359f4df5a3b1922bd698633e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 17:00:40 UTC) from {username='slave'} at 127.0.0.1:32890
I20260619 17:00:40.305510 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4071
I20260619 17:00:40.316936 4506 tablet_replica.cc:330] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: stopping tablet replica
I20260619 17:00:40.317781 4507 tablet_replica.cc:330] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: stopping tablet replica
I20260619 17:00:40.317896 4506 raft_consensus.cc:2229] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 1 LEADER]: Raft consensus shutting down.
I20260619 17:00:40.318554 4508 tablet_replica.cc:330] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: stopping tablet replica
I20260619 17:00:40.319187 4506 raft_consensus.cc:2258] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:40.319533 4507 raft_consensus.cc:2229] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:40.320096 4507 raft_consensus.cc:2258] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:40.320266 4508 raft_consensus.cc:2229] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 17:00:40.320861 4508 raft_consensus.cc:2258] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 17:00:40.321627 4506 ts_tablet_manager.cc:1905] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:40.323196 4507 ts_tablet_manager.cc:1905] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:40.323194 4508 ts_tablet_manager.cc:1905] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 17:00:40.330091 4506 ts_tablet_manager.cc:1918] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:40.330461 4506 log.cc:1196] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-1/wal/wals/2b6233a6359f4df5a3b1922bd698633e
I20260619 17:00:40.330440 4507 ts_tablet_manager.cc:1918] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:40.330605 4507 log.cc:1196] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-2/wal/wals/2b6233a6359f4df5a3b1922bd698633e
I20260619 17:00:40.331147 4508 ts_tablet_manager.cc:1918] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 17:00:40.331210 4507 ts_tablet_manager.cc:1939] T 2b6233a6359f4df5a3b1922bd698633e P a9bcb898ce2a4705810406ca8081fcd5: Deleting consensus metadata
I20260619 17:00:40.331223 4506 ts_tablet_manager.cc:1939] T 2b6233a6359f4df5a3b1922bd698633e P 708919e6fbfa425f8a8988fddbf1f0ff: Deleting consensus metadata
I20260619 17:00:40.333261 3996 catalog_manager.cc:4841] TS a9bcb898ce2a4705810406ca8081fcd5 (127.16.48.3:45217): tablet 2b6233a6359f4df5a3b1922bd698633e (table test-table [id=9f45428438924a48a76e33db680781dc]) successfully deleted
I20260619 17:00:40.333262 3999 catalog_manager.cc:4841] TS 708919e6fbfa425f8a8988fddbf1f0ff (127.16.48.2:40813): tablet 2b6233a6359f4df5a3b1922bd698633e (table test-table [id=9f45428438924a48a76e33db680781dc]) successfully deleted
I20260619 17:00:40.333837 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.307324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.16.48.1:40975)
0619 17:00:40.307519 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:40.307526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:40.313273 (+ 5747us) server_negotiation.cc:407] Connection header received
0619 17:00:40.313524 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:40.313530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:40.313655 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:40.313791 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:40.316532 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.317765 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.320313 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.321075 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.321195 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.331868 (+ 10673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:40.331884 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:40.331916 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:40.332763 (+ 847us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:40.332845 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:40.332876 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:40.333015 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:40.333306 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 17:00:40.333450 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 17:00:40.333846 4508 log.cc:1196] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: Deleting WAL directory at /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781888107067647-16576-0/minicluster-data/ts-0/wal/wals/2b6233a6359f4df5a3b1922bd698633e
I20260619 17:00:40.334544 4508 ts_tablet_manager.cc:1939] T 2b6233a6359f4df5a3b1922bd698633e P 0e3dde2745c24c49b1cbf92350b2cf4e: Deleting consensus metadata
I20260619 17:00:40.335063 4193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:40.445468 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 4071
I20260619 17:00:40.446269 3997 catalog_manager.cc:4841] TS 0e3dde2745c24c49b1cbf92350b2cf4e (127.16.48.1:40975): tablet 2b6233a6359f4df5a3b1922bd698633e (table test-table [id=9f45428438924a48a76e33db680781dc]) successfully deleted
I20260619 17:00:40.464670 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4202
I20260619 17:00:40.472733 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.465506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.16.48.2:40813)
0619 17:00:40.465669 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:40.465674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:40.465693 (+ 19us) server_negotiation.cc:407] Connection header received
0619 17:00:40.465894 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:40.465898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:40.466011 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:40.466097 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:40.466580 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.467294 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.469348 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.469835 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.469955 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.471179 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:40.471187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:40.471203 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:40.471856 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:40.471918 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:40.471940 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:40.472022 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:40.472263 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 17:00:40.472363 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 17:00:40.473430 4324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:40.589663 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 4202
I20260619 17:00:40.610122 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4332
I20260619 17:00:40.617764 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:40.610971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.16.48.3:45217)
0619 17:00:40.611149 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:40.611154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:40.611234 (+ 80us) server_negotiation.cc:407] Connection header received
0619 17:00:40.611452 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:40.611458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:40.611575 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:40.611698 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:40.612159 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.612930 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.614290 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:40.614768 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:40.614918 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:40.616212 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:40.616222 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:40.616237 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:40.616878 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:40.616929 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:40.616952 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:40.617089 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:40.617248 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 17:00:40.617360 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 17:00:40.618502 4454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:40.713832 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 4332
I20260619 17:00:40.731148 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 3981
I20260619 17:00:40.732388 4042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:40.843864 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 3981
2026-06-19T17:00:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3416 ms)
[----------] 6 tests from EncryptionPolicyTest (23773 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:40 dist-test-slave-3ffd krb5kdc[4516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:40 dist-test-slave-3ffd krb5kdc[4516](info): set up 2 sockets
Jun 19 17:00:40 dist-test-slave-3ffd krb5kdc[4516](info): commencing operation
krb5kdc: starting...
W20260619 17:00:41.977003 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:42 dist-test-slave-3ffd krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:00:42.257812 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:42.620322 4529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:42.620708 4529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:42.620867 4529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:42.628670 4529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:42.628849 4529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:42.628959 4529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:42.629079 4529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:42.629158 4529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:42.639742 4529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:42.641301 4529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:42.643019 4529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:42.656222 4535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:42.656270 4537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:42.656188 4534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:42.657590 4529 server_base.cc:1034] running on GCE node
I20260619 17:00:42.658459 4529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:42.660071 4529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:42.661384 4529 hybrid_clock.cc:648] HybridClock initialized: now 1781888442661329 us; error 54 us; skew 500 ppm
I20260619 17:00:42.661798 4529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:42 dist-test-slave-3ffd krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:42.671190 4529 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:00:42.673216 4529 webserver.cc:460] Webserver started at http://127.0.0.1:38435/ using document root <none> and password file <none>
I20260619 17:00:42.674047 4529 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:42.674226 4529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:42.674598 4529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:42.678057 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "b1758fc30037437abf39b2b8cc3218ff"
format_stamp: "Formatted at 2026-06-19 17:00:42 on dist-test-slave-3ffd"
server_key: "06a62fb4001548e0f4abc0939c8069c9"
server_key_iv: "495b438946161417603e39fd97dd534e"
server_key_version: "encryptionkey@0"
I20260619 17:00:42.679041 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "b1758fc30037437abf39b2b8cc3218ff"
format_stamp: "Formatted at 2026-06-19 17:00:42 on dist-test-slave-3ffd"
server_key: "06a62fb4001548e0f4abc0939c8069c9"
server_key_iv: "495b438946161417603e39fd97dd534e"
server_key_version: "encryptionkey@0"
I20260619 17:00:42.685628 4529 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 17:00:42.690235 4544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:42.692067 4529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 17:00:42.692509 4529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "b1758fc30037437abf39b2b8cc3218ff"
format_stamp: "Formatted at 2026-06-19 17:00:42 on dist-test-slave-3ffd"
server_key: "06a62fb4001548e0f4abc0939c8069c9"
server_key_iv: "495b438946161417603e39fd97dd534e"
server_key_version: "encryptionkey@0"
I20260619 17:00:42.692844 4529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:42.718180 4529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:42.723205 4529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:42.723542 4529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:42.743781 4529 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33299
I20260619 17:00:42.743813 4595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33299 every 8 connection(s)
I20260619 17:00:42.745477 4529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:42.746003 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 4529
I20260619 17:00:42.746239 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:42.747076 16576 external_mini_cluster.cc:1428] Setting key 2c8c059e2a3f62cade81eab9b6aa43e3
I20260619 17:00:42.750607 4596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 17:00:42 dist-test-slave-3ffd krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 17:00:42.765491 4596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff: Bootstrap starting.
I20260619 17:00:42.769980 4596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:42.771607 4596 log.cc:824] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:42.774434 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:42.749425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49507 (local address 127.0.0.1:33299)
0619 17:00:42.750285 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:42.750301 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:42.750361 (+ 60us) server_negotiation.cc:407] Connection header received
0619 17:00:42.752857 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:42.752890 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:42.753505 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:42.754008 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:42.755491 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:42.757762 (+ 2271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:42.759249 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:42.760410 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:42.760622 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:42.765384 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:42.765395 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:42.765413 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:42.765466 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:42.769051 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:42.770335 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:42.770343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:42.770350 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:42.770617 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:42.771280 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:42.771283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:42.771285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:42.771968 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:42.772345 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:42.773004 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 17:00:42.773366 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":348,"threads_started":1}
I20260619 17:00:42.775138 4596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff: No bootstrap required, opened a new log
I20260619 17:00:42.779354 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:00:42.784492 4596 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1758fc30037437abf39b2b8cc3218ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33299 } }
I20260619 17:00:42.785037 4596 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:42.785197 4596 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1758fc30037437abf39b2b8cc3218ff, State: Initialized, Role: FOLLOWER
I20260619 17:00:42.785933 4596 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1758fc30037437abf39b2b8cc3218ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33299 } }
I20260619 17:00:42.786275 4596 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:42.786428 4596 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:42.786644 4596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:42.789942 4596 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1758fc30037437abf39b2b8cc3218ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33299 } }
I20260619 17:00:42.790704 4596 leader_election.cc:302] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1758fc30037437abf39b2b8cc3218ff; no voters:
I20260619 17:00:42.791513 4596 leader_election.cc:288] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:42.791586 4606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:42.792866 4606 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [term 1 LEADER]: Becoming Leader. State: Replica: b1758fc30037437abf39b2b8cc3218ff, State: Running, Role: LEADER
I20260619 17:00:42.793630 4606 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1758fc30037437abf39b2b8cc3218ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33299 } }
I20260619 17:00:42.797887 4596 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:42.798226 4607 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1758fc30037437abf39b2b8cc3218ff. Latest consensus state: current_term: 1 leader_uuid: "b1758fc30037437abf39b2b8cc3218ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1758fc30037437abf39b2b8cc3218ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33299 } } }
I20260619 17:00:42.798694 4607 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:42.804447 4608 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1758fc30037437abf39b2b8cc3218ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1758fc30037437abf39b2b8cc3218ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33299 } } }
I20260619 17:00:42.804857 4608 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:42.806200 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:42.784481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55165 (local address 127.0.0.1:33299)
0619 17:00:42.784695 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:42.784701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:42.785161 (+ 460us) server_negotiation.cc:407] Connection header received
0619 17:00:42.785373 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:42.785381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:42.785455 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:42.785586 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:42.787583 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:42.788543 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:42.790431 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:42.790966 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:42.791107 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:42.795195 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:42.795206 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:42.795217 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:42.795285 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:42.800506 (+ 5221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:42.802213 (+ 1707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:42.802222 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:42.802228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:42.802547 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:42.803338 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:42.803351 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:42.803357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:42.804491 (+ 1134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:42.804871 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:42.805384 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 17:00:42.805667 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 17:00:42.809569 4610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:42.814672 4610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:42.823740 4610 catalog_manager.cc:1347] Generated new cluster ID: 87beb6a6ef1e4372b7dd81a023ed5940
I20260619 17:00:42.823894 4610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:42.853189 4610 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:42.854609 4610 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:42.864732 4610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1758fc30037437abf39b2b8cc3218ff: Generated new TSK 0
I20260619 17:00:42.865725 4610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:42.879494 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4529
I20260619 17:00:42.880991 4591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:42.977886 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 4529
W20260619 17:00:42.996075 4514 connection.cc:535] client connection to 127.0.0.1:33299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33299): Connection reset by peer (error 104)
2026-06-19T17:00:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:43 dist-test-slave-3ffd krb5kdc[4628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:43 dist-test-slave-3ffd krb5kdc[4628](info): set up 2 sockets
Jun 19 17:00:43 dist-test-slave-3ffd krb5kdc[4628](info): commencing operation
krb5kdc: starting...
W20260619 17:00:44.108878 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:44 dist-test-slave-3ffd krb5kdc[4628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:00:44.384783 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:44.737653 4641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:44.738055 4641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:44.738222 4641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:44.746049 4641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:44.746245 4641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:44.746366 4641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:44.746487 4641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:44.746568 4641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:44.757186 4641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:44.758695 4641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:44.760169 4641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:44.773334 4646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:44.773437 4649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:44.773379 4647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:44.773820 4641 server_base.cc:1034] running on GCE node
I20260619 17:00:44.775068 4641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:44.776768 4641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:44.778045 4641 hybrid_clock.cc:648] HybridClock initialized: now 1781888444777983 us; error 56 us; skew 500 ppm
I20260619 17:00:44.778450 4641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:44 dist-test-slave-3ffd krb5kdc[4628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:44.782029 4641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:00:44.783736 4641 webserver.cc:460] Webserver started at http://127.0.0.1:38545/ using document root <none> and password file <none>
I20260619 17:00:44.784581 4641 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:44.784744 4641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:44.785072 4641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:44.788470 4641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1"
format_stamp: "Formatted at 2026-06-19 17:00:44 on dist-test-slave-3ffd"
server_key: "4240fdac51750abf7467e7ebfafe1dd1"
server_key_iv: "30cb0bd005b7c653a129e06ac17fd1c2"
server_key_version: "encryptionkey@0"
I20260619 17:00:44.789356 4641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1"
format_stamp: "Formatted at 2026-06-19 17:00:44 on dist-test-slave-3ffd"
server_key: "4240fdac51750abf7467e7ebfafe1dd1"
server_key_iv: "30cb0bd005b7c653a129e06ac17fd1c2"
server_key_version: "encryptionkey@0"
I20260619 17:00:44.795251 4641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 17:00:44.799926 4656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:44.801666 4641 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 17:00:44.801904 4641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1"
format_stamp: "Formatted at 2026-06-19 17:00:44 on dist-test-slave-3ffd"
server_key: "4240fdac51750abf7467e7ebfafe1dd1"
server_key_iv: "30cb0bd005b7c653a129e06ac17fd1c2"
server_key_version: "encryptionkey@0"
I20260619 17:00:44.802150 4641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:44.826525 4641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:44.831183 4641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:44.831530 4641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:44.852219 4641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42067
I20260619 17:00:44.852273 4707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42067 every 8 connection(s)
I20260619 17:00:44.853739 4641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:44.857808 4708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:44.862319 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 4641
I20260619 17:00:44.862550 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:44.862982 16576 external_mini_cluster.cc:1428] Setting key 686ad7867b5f20955e4dcdc1d0d437fb
I20260619 17:00:44.871313 4708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1: Bootstrap starting.
I20260619 17:00:44.875298 4708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:44.876770 4708 log.cc:824] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:44.879686 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:44.865048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40351 (local address 127.0.0.1:42067)
0619 17:00:44.865956 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:44.865974 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:44.866028 (+ 54us) server_negotiation.cc:407] Connection header received
0619 17:00:44.869202 (+ 3174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:44.869241 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:44.869870 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:44.870364 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:44.870935 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:44.873029 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:44.874523 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:44.875621 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:44.875838 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:44.877135 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:44.877147 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:44.877166 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:44.877891 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:44.877980 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:44.878011 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:44.878198 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:44.878510 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 17:00:44.878684 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":420,"threads_started":1}
I20260619 17:00:44.880247 4708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1: No bootstrap required, opened a new log
I20260619 17:00:44.884241 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:00:44.888727 4708 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260619 17:00:44.889274 4708 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:44.889405 4708 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e55cb10f4c4ad7b6fd6db806ec4ab1, State: Initialized, Role: FOLLOWER
I20260619 17:00:44.890178 4708 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260619 17:00:44.890450 4708 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:44.890635 4708 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:44.890842 4708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:44.893517 4708 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260619 17:00:44.894150 4708 leader_election.cc:302] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5e55cb10f4c4ad7b6fd6db806ec4ab1; no voters:
I20260619 17:00:44.894829 4708 leader_election.cc:288] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:44.894958 4718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:44.896718 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:44.888769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35971 (local address 127.0.0.1:42067)
0619 17:00:44.888968 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:44.888974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:44.889487 (+ 513us) server_negotiation.cc:407] Connection header received
0619 17:00:44.889699 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:44.889704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:44.889794 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:44.889917 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:44.890551 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:44.891455 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:44.893067 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:44.893561 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:44.893707 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:44.895257 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:44.895269 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:44.895281 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:44.895316 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:44.895395 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:44.895434 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:44.895586 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:44.896310 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 17:00:44.896370 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 17:00:44.897518 4708 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:44.899547 4718 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [term 1 LEADER]: Becoming Leader. State: Replica: b5e55cb10f4c4ad7b6fd6db806ec4ab1, State: Running, Role: LEADER
I20260619 17:00:44.900411 4718 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260619 17:00:44.904043 4720 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5e55cb10f4c4ad7b6fd6db806ec4ab1. Latest consensus state: current_term: 1 leader_uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } } }
I20260619 17:00:44.904616 4720 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:44.904023 4719 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e55cb10f4c4ad7b6fd6db806ec4ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } } }
I20260619 17:00:44.905069 4719 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:44.905560 4729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:44.912247 4729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:44.922490 4729 catalog_manager.cc:1347] Generated new cluster ID: 4d4df7b4df7d4f6cb1db1aac67e101be
I20260619 17:00:44.922680 4729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:44.936722 4729 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:44.938699 4729 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:44.953219 4729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5e55cb10f4c4ad7b6fd6db806ec4ab1: Generated new TSK 0
I20260619 17:00:44.954680 4729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:44.972994 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4641
W20260619 17:00:44.974860 4703 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40351
W20260619 17:00:44.975785 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4641: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:00:44.976011 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 4641
W20260619 17:00:45.004539 4626 connection.cc:535] client connection to 127.0.0.1:42067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42067): Connection reset by peer (error 104)
2026-06-19T17:00:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2008 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:45 dist-test-slave-3ffd krb5kdc[4739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:45 dist-test-slave-3ffd krb5kdc[4739](info): set up 2 sockets
Jun 19 17:00:45 dist-test-slave-3ffd krb5kdc[4739](info): commencing operation
krb5kdc: starting...
W20260619 17:00:46.104969 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:46 dist-test-slave-3ffd krb5kdc[4739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:00:46.379751 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:46.734915 4753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:46.735255 4753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:46.735354 4753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:46.743034 4753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:46.743176 4753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:46.743223 4753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:46.743279 4753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:46.743324 4753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:46.753844 4753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:46.755190 4753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:46.756546 4753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:46.769755 4758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:46.769755 4759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:46.769784 4761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:46.771090 4753 server_base.cc:1034] running on GCE node
I20260619 17:00:46.771981 4753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:46.773600 4753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:46.774879 4753 hybrid_clock.cc:648] HybridClock initialized: now 1781888446774839 us; error 62 us; skew 500 ppm
I20260619 17:00:46.775286 4753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:46 dist-test-slave-3ffd krb5kdc[4739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:46.778753 4753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:00:46.780613 4753 webserver.cc:460] Webserver started at http://127.0.0.1:43641/ using document root <none> and password file <none>
I20260619 17:00:46.781397 4753 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:46.781577 4753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:46.781912 4753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:46.785359 4753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "3050075c77a34d7eab708fcce7bb133c"
format_stamp: "Formatted at 2026-06-19 17:00:46 on dist-test-slave-3ffd"
server_key: "5d5d8f186166ff8d6e689d1f54519c82"
server_key_iv: "73a53904f2d2fe24496f2bfeb8cb5312"
server_key_version: "encryptionkey@0"
I20260619 17:00:46.786296 4753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "3050075c77a34d7eab708fcce7bb133c"
format_stamp: "Formatted at 2026-06-19 17:00:46 on dist-test-slave-3ffd"
server_key: "5d5d8f186166ff8d6e689d1f54519c82"
server_key_iv: "73a53904f2d2fe24496f2bfeb8cb5312"
server_key_version: "encryptionkey@0"
I20260619 17:00:46.792622 4753 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 17:00:46.796860 4768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:46.798656 4753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 17:00:46.798924 4753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "3050075c77a34d7eab708fcce7bb133c"
format_stamp: "Formatted at 2026-06-19 17:00:46 on dist-test-slave-3ffd"
server_key: "5d5d8f186166ff8d6e689d1f54519c82"
server_key_iv: "73a53904f2d2fe24496f2bfeb8cb5312"
server_key_version: "encryptionkey@0"
I20260619 17:00:46.799201 4753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:46.805991 4753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:46.810971 4753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:46.811353 4753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:46.832465 4753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38867
I20260619 17:00:46.832571 4819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38867 every 8 connection(s)
I20260619 17:00:46.834017 4753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:46.837152 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 4753
I20260619 17:00:46.837343 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:46.837888 16576 external_mini_cluster.cc:1428] Setting key 7777a5324b4cd5a74442b7357e7bb6a8
I20260619 17:00:46.838762 4820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:46.846594 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:46.839913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55741 (local address 127.0.0.1:38867)
0619 17:00:46.840874 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:46.840887 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:46.840931 (+ 44us) server_negotiation.cc:407] Connection header received
0619 17:00:46.842545 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:46.842570 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:46.842980 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:46.843379 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:46.844760 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:46.844766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:46.844782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:46.844804 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:46.844886 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:46.844913 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:46.845123 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:46.845282 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 17:00:46.845531 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":466,"threads_started":1}
I20260619 17:00:46.850754 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:00:46.855026 4820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c: Bootstrap starting.
I20260619 17:00:46.859730 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:46.856884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41981 (local address 127.0.0.1:38867)
0619 17:00:46.857129 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:46.857135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:46.857562 (+ 427us) server_negotiation.cc:407] Connection header received
0619 17:00:46.857778 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:46.857782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:46.857836 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:46.857930 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:46.858984 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:46.858989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:46.858992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:46.859008 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:46.859049 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:46.859065 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:46.859153 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:46.859320 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 17:00:46.859373 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 17:00:46.859994 4820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:46.861627 4820 log.cc:824] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:46.864970 4820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c: No bootstrap required, opened a new log
I20260619 17:00:46.872953 4820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050075c77a34d7eab708fcce7bb133c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } }
I20260619 17:00:46.873410 4820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:46.873538 4820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3050075c77a34d7eab708fcce7bb133c, State: Initialized, Role: FOLLOWER
I20260619 17:00:46.874184 4820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050075c77a34d7eab708fcce7bb133c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } }
I20260619 17:00:46.874428 4820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:46.874629 4820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:46.874815 4820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:46.877259 4820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050075c77a34d7eab708fcce7bb133c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } }
I20260619 17:00:46.877843 4820 leader_election.cc:302] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3050075c77a34d7eab708fcce7bb133c; no voters:
I20260619 17:00:46.878584 4820 leader_election.cc:288] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:46.878773 4830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:46.879801 4830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [term 1 LEADER]: Becoming Leader. State: Replica: 3050075c77a34d7eab708fcce7bb133c, State: Running, Role: LEADER
I20260619 17:00:46.880489 4830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050075c77a34d7eab708fcce7bb133c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } }
I20260619 17:00:46.881186 4820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:46.883417 4831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3050075c77a34d7eab708fcce7bb133c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050075c77a34d7eab708fcce7bb133c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } } }
I20260619 17:00:46.884081 4831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:46.884057 4832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3050075c77a34d7eab708fcce7bb133c. Latest consensus state: current_term: 1 leader_uuid: "3050075c77a34d7eab708fcce7bb133c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050075c77a34d7eab708fcce7bb133c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } } }
I20260619 17:00:46.884407 4832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:46.887104 4837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:46.893198 4837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:46.904162 4837 catalog_manager.cc:1347] Generated new cluster ID: 55139d9cce5d4e0fb0b651c481cc212a
I20260619 17:00:46.904305 4837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:46.918639 4837 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:46.919205 4837 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:46.927850 4837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3050075c77a34d7eab708fcce7bb133c: Generated new TSK 0
I20260619 17:00:46.928882 4837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:46.944520 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4753
W20260619 17:00:46.945771 4815 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55741
W20260619 17:00:46.946530 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4753: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:00:46.946694 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 4753
2026-06-19T17:00:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:47 dist-test-slave-3ffd krb5kdc[4851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:47 dist-test-slave-3ffd krb5kdc[4851](info): set up 2 sockets
Jun 19 17:00:47 dist-test-slave-3ffd krb5kdc[4851](info): commencing operation
krb5kdc: starting...
W20260619 17:00:48.057367 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:48 dist-test-slave-3ffd krb5kdc[4851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:00:48.340428 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:48.694705 4864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:48.695036 4864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:48.695135 4864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:48.706350 4864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:48.706501 4864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:48.706549 4864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:48.706612 4864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:48.706657 4864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:48.717880 4864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:48.719262 4864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:48.720719 4864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:48.733464 4869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:48.733567 4870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:48.733624 4872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:48.734370 4864 server_base.cc:1034] running on GCE node
I20260619 17:00:48.735276 4864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:48.736941 4864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:48.738169 4864 hybrid_clock.cc:648] HybridClock initialized: now 1781888448738123 us; error 38 us; skew 500 ppm
I20260619 17:00:48.738580 4864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:48 dist-test-slave-3ffd krb5kdc[4851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:48.748730 4864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:00:48.750708 4864 webserver.cc:460] Webserver started at http://127.0.0.1:42169/ using document root <none> and password file <none>
I20260619 17:00:48.751559 4864 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:48.751750 4864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:48.752190 4864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:48.755834 4864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "4f4fd59ef5d344ba83935614be72b0fd"
format_stamp: "Formatted at 2026-06-19 17:00:48 on dist-test-slave-3ffd"
server_key: "3f010786d0c2855c79f3b6d857bb8258"
server_key_iv: "0ebf9fa07f3cb3bb5a49d6afdf5e9a1a"
server_key_version: "encryptionkey@0"
I20260619 17:00:48.757079 4864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "4f4fd59ef5d344ba83935614be72b0fd"
format_stamp: "Formatted at 2026-06-19 17:00:48 on dist-test-slave-3ffd"
server_key: "3f010786d0c2855c79f3b6d857bb8258"
server_key_iv: "0ebf9fa07f3cb3bb5a49d6afdf5e9a1a"
server_key_version: "encryptionkey@0"
I20260619 17:00:48.763643 4864 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 17:00:48.768059 4879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:48.769711 4864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 17:00:48.769896 4864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "4f4fd59ef5d344ba83935614be72b0fd"
format_stamp: "Formatted at 2026-06-19 17:00:48 on dist-test-slave-3ffd"
server_key: "3f010786d0c2855c79f3b6d857bb8258"
server_key_iv: "0ebf9fa07f3cb3bb5a49d6afdf5e9a1a"
server_key_version: "encryptionkey@0"
I20260619 17:00:48.770102 4864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:48.789603 4864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:48.794484 4864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:48.794819 4864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:48.815511 4864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40139
I20260619 17:00:48.815557 4930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40139 every 8 connection(s)
I20260619 17:00:48.817134 4864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:48.819566 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 4864
I20260619 17:00:48.819717 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:48.820286 16576 external_mini_cluster.cc:1428] Setting key 152b2dacfae8af7653d99cf27d91a872
I20260619 17:00:48.822163 4931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 17:00:48 dist-test-slave-3ffd krb5kdc[4851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 17:00:48.838811 4931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd: Bootstrap starting.
I20260619 17:00:48.842721 4931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:48.844071 4931 log.cc:824] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:48.846951 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:48.822418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49617 (local address 127.0.0.1:40139)
0619 17:00:48.823315 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:48.823327 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:48.823371 (+ 44us) server_negotiation.cc:407] Connection header received
0619 17:00:48.825153 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:48.825180 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:48.825654 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:48.826046 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:48.828301 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:48.829792 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:48.831978 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:48.832873 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:48.833049 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:48.838666 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:48.838676 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:48.838695 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:48.838744 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:48.842265 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:48.843208 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:48.843220 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:48.843227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:48.843419 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:48.843858 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:48.843863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:48.843869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:48.844545 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:48.844865 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:48.845377 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 17:00:48.845931 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":418,"threads_started":1}
I20260619 17:00:48.847647 4931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd: No bootstrap required, opened a new log
I20260619 17:00:48.851008 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:00:48.857177 4931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fd59ef5d344ba83935614be72b0fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } }
I20260619 17:00:48.857677 4931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:48.857824 4931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4fd59ef5d344ba83935614be72b0fd, State: Initialized, Role: FOLLOWER
I20260619 17:00:48.858487 4931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fd59ef5d344ba83935614be72b0fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } }
I20260619 17:00:48.858800 4931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:48.858986 4931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:48.859220 4931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:48.861927 4931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fd59ef5d344ba83935614be72b0fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } }
I20260619 17:00:48.862564 4931 leader_election.cc:302] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f4fd59ef5d344ba83935614be72b0fd; no voters:
I20260619 17:00:48.863287 4931 leader_election.cc:288] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:48.863451 4941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:48.864781 4941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [term 1 LEADER]: Becoming Leader. State: Replica: 4f4fd59ef5d344ba83935614be72b0fd, State: Running, Role: LEADER
I20260619 17:00:48.865528 4941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fd59ef5d344ba83935614be72b0fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } }
I20260619 17:00:48.866142 4931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:48.868618 4943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f4fd59ef5d344ba83935614be72b0fd. Latest consensus state: current_term: 1 leader_uuid: "4f4fd59ef5d344ba83935614be72b0fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fd59ef5d344ba83935614be72b0fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } } }
I20260619 17:00:48.869110 4943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:48.869514 4942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f4fd59ef5d344ba83935614be72b0fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fd59ef5d344ba83935614be72b0fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } } }
I20260619 17:00:48.869948 4942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:48.876101 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:48.855728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56311 (local address 127.0.0.1:40139)
0619 17:00:48.855904 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:48.855909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:48.856442 (+ 533us) server_negotiation.cc:407] Connection header received
0619 17:00:48.856660 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:48.856675 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:48.856756 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:48.856870 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:48.858608 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:48.859670 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:48.861128 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:48.861599 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:48.861753 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:48.865274 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:48.865284 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:48.865289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:48.865375 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:48.870382 (+ 5007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:48.871674 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:48.871683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:48.871689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:48.871979 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:48.872693 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:48.872701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:48.872707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:48.873226 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:48.875008 (+ 1782us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:48.875218 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 17:00:48.875532 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 17:00:48.881567 4946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:48.887413 4946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:48.897359 4946 catalog_manager.cc:1347] Generated new cluster ID: 10f5f7a7692144359aaf29d6dafd9519
I20260619 17:00:48.897509 4946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:48.942493 4946 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:48.943953 4946 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:48.951210 4946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f4fd59ef5d344ba83935614be72b0fd: Generated new TSK 0
I20260619 17:00:48.952317 4946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:48.965945 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4864
I20260619 17:00:48.967262 4926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:49.064654 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 4864
2026-06-19T17:00:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:49 dist-test-slave-3ffd krb5kdc[4963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:49 dist-test-slave-3ffd krb5kdc[4963](info): set up 2 sockets
Jun 19 17:00:49 dist-test-slave-3ffd krb5kdc[4963](info): commencing operation
krb5kdc: starting...
W20260619 17:00:50.192864 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:50 dist-test-slave-3ffd krb5kdc[4963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:00:50.467548 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:50.834197 4976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:50.834538 4976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:50.834636 4976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:50.842211 4976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:50.842347 4976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:50.842396 4976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:50.842463 4976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:50.842510 4976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:50.853055 4976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:50.854447 4976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:50.855902 4976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:50.868988 4981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:50.868965 4982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:50.869047 4984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:50.870483 4976 server_base.cc:1034] running on GCE node
I20260619 17:00:50.871476 4976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:50.873219 4976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:50.874491 4976 hybrid_clock.cc:648] HybridClock initialized: now 1781888450874415 us; error 58 us; skew 500 ppm
I20260619 17:00:50.874914 4976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:50 dist-test-slave-3ffd krb5kdc[4963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:50.878409 4976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:00:50.880020 4976 webserver.cc:460] Webserver started at http://127.0.0.1:37769/ using document root <none> and password file <none>
I20260619 17:00:50.880817 4976 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:50.880929 4976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:50.881202 4976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:50.884545 4976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "d849caa6dd7f4f089bb76b48d88edb85"
format_stamp: "Formatted at 2026-06-19 17:00:50 on dist-test-slave-3ffd"
server_key: "97997d8ed98bd4028f744ad99fa3ce61"
server_key_iv: "021c0bb52238e47bb2bba38f7adac645"
server_key_version: "encryptionkey@0"
I20260619 17:00:50.885403 4976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "d849caa6dd7f4f089bb76b48d88edb85"
format_stamp: "Formatted at 2026-06-19 17:00:50 on dist-test-slave-3ffd"
server_key: "97997d8ed98bd4028f744ad99fa3ce61"
server_key_iv: "021c0bb52238e47bb2bba38f7adac645"
server_key_version: "encryptionkey@0"
I20260619 17:00:50.891235 4976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 17:00:50.895385 4991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:50.897121 4976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 17:00:50.897341 4976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "d849caa6dd7f4f089bb76b48d88edb85"
format_stamp: "Formatted at 2026-06-19 17:00:50 on dist-test-slave-3ffd"
server_key: "97997d8ed98bd4028f744ad99fa3ce61"
server_key_iv: "021c0bb52238e47bb2bba38f7adac645"
server_key_version: "encryptionkey@0"
I20260619 17:00:50.897550 4976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:50.912288 4976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:50.917418 4976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:50.917764 4976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:50.938102 4976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33231
I20260619 17:00:50.938138 5042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33231 every 8 connection(s)
I20260619 17:00:50.939610 4976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:50.943408 5043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:50.946617 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 4976
I20260619 17:00:50.946805 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:50.947243 16576 external_mini_cluster.cc:1428] Setting key bdb357a4f3a1fe28a55e60f3b589e44b
I20260619 17:00:50.956251 5043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85: Bootstrap starting.
I20260619 17:00:50.960606 5043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:50.961761 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:50.949291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51993 (local address 127.0.0.1:33231)
0619 17:00:50.950141 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:50.950153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:50.950199 (+ 46us) server_negotiation.cc:407] Connection header received
0619 17:00:50.951985 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:50.952009 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:50.952564 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:50.952955 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:50.953608 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:50.954941 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:50.956851 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:50.957540 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:50.957697 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:50.959123 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:50.959131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:50.959147 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:50.959744 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:50.959825 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:50.959853 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:50.959950 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:50.960281 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 17:00:50.960637 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":409,"threads_started":1}
I20260619 17:00:50.961866 5043 log.cc:824] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:50.965579 5043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85: No bootstrap required, opened a new log
I20260619 17:00:50.966255 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:00:50.974093 5043 raft_consensus.cc:348] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d849caa6dd7f4f089bb76b48d88edb85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } }
I20260619 17:00:50.974543 5043 raft_consensus.cc:374] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:50.974673 5043 raft_consensus.cc:729] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d849caa6dd7f4f089bb76b48d88edb85, State: Initialized, Role: FOLLOWER
I20260619 17:00:50.975340 5043 consensus_queue.cc:260] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d849caa6dd7f4f089bb76b48d88edb85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } }
I20260619 17:00:50.975621 5043 raft_consensus.cc:388] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:50.975790 5043 raft_consensus.cc:482] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:50.975967 5043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:50.977551 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:50.970713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33417 (local address 127.0.0.1:33231)
0619 17:00:50.970973 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:50.970983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:50.971536 (+ 553us) server_negotiation.cc:407] Connection header received
0619 17:00:50.971780 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:50.971784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:50.971851 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:50.971976 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:50.972556 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:50.973316 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:50.974756 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:50.975357 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:50.975501 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:50.976818 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:50.976823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:50.976826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:50.976851 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:50.976895 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:50.976914 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:50.977023 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:50.977139 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 17:00:50.977245 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 17:00:50.979002 5043 raft_consensus.cc:504] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d849caa6dd7f4f089bb76b48d88edb85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } }
I20260619 17:00:50.979593 5043 leader_election.cc:302] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d849caa6dd7f4f089bb76b48d88edb85; no voters:
I20260619 17:00:50.980248 5043 leader_election.cc:288] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:50.980386 5053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:50.981225 5053 raft_consensus.cc:686] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [term 1 LEADER]: Becoming Leader. State: Replica: d849caa6dd7f4f089bb76b48d88edb85, State: Running, Role: LEADER
I20260619 17:00:50.981832 5053 consensus_queue.cc:237] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d849caa6dd7f4f089bb76b48d88edb85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } }
I20260619 17:00:50.982771 5043 sys_catalog.cc:564] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:50.985433 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d849caa6dd7f4f089bb76b48d88edb85. Latest consensus state: current_term: 1 leader_uuid: "d849caa6dd7f4f089bb76b48d88edb85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d849caa6dd7f4f089bb76b48d88edb85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } } }
I20260619 17:00:50.985425 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d849caa6dd7f4f089bb76b48d88edb85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d849caa6dd7f4f089bb76b48d88edb85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } } }
I20260619 17:00:50.985901 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:50.985918 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:50.990475 5058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:50.996694 5058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:51.007236 5058 catalog_manager.cc:1347] Generated new cluster ID: 52180e40fbaf4dbf8e680e7419503f07
I20260619 17:00:51.007392 5058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:51.032960 5058 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:51.035149 5058 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:51.061808 5058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d849caa6dd7f4f089bb76b48d88edb85: Generated new TSK 0
I20260619 17:00:51.063151 5058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:51.077898 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4976
W20260619 17:00:51.079186 5038 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51993
W20260619 17:00:51.080065 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 4976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:00:51.080291 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 4976
2026-06-19T17:00:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:51 dist-test-slave-3ffd krb5kdc[5074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:51 dist-test-slave-3ffd krb5kdc[5074](info): set up 2 sockets
Jun 19 17:00:51 dist-test-slave-3ffd krb5kdc[5074](info): commencing operation
krb5kdc: starting...
W20260619 17:00:52.205132 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:52 dist-test-slave-3ffd krb5kdc[5074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:00:52.483861 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:52.843905 5087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:52.844360 5087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:52.844532 5087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:52.852353 5087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:52.852490 5087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:52.852576 5087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:52.852689 5087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:52.852756 5087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:52.863548 5087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:52.865072 5087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:52.866531 5087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:52.880223 5092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:52.880191 5095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:52.880296 5093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:52.881196 5087 server_base.cc:1034] running on GCE node
I20260619 17:00:52.882089 5087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:52.883704 5087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:52.884958 5087 hybrid_clock.cc:648] HybridClock initialized: now 1781888452884926 us; error 64 us; skew 500 ppm
I20260619 17:00:52.885365 5087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:52 dist-test-slave-3ffd krb5kdc[5074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:52.888805 5087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:00:52.890662 5087 webserver.cc:460] Webserver started at http://127.0.0.1:39021/ using document root <none> and password file <none>
I20260619 17:00:52.891449 5087 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:52.891618 5087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:52.891953 5087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:52.895313 5087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3"
format_stamp: "Formatted at 2026-06-19 17:00:52 on dist-test-slave-3ffd"
server_key: "cf6aa1f1aca5c0d47ca8ff9ccec6788a"
server_key_iv: "99aa4b99f9038bc929f047d01eb0b7b6"
server_key_version: "encryptionkey@0"
I20260619 17:00:52.896273 5087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3"
format_stamp: "Formatted at 2026-06-19 17:00:52 on dist-test-slave-3ffd"
server_key: "cf6aa1f1aca5c0d47ca8ff9ccec6788a"
server_key_iv: "99aa4b99f9038bc929f047d01eb0b7b6"
server_key_version: "encryptionkey@0"
I20260619 17:00:52.902072 5087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 17:00:52.906196 5102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:52.907753 5087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 17:00:52.907994 5087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3"
format_stamp: "Formatted at 2026-06-19 17:00:52 on dist-test-slave-3ffd"
server_key: "cf6aa1f1aca5c0d47ca8ff9ccec6788a"
server_key_iv: "99aa4b99f9038bc929f047d01eb0b7b6"
server_key_version: "encryptionkey@0"
I20260619 17:00:52.908309 5087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:52.914898 5087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:52.919701 5087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:52.920059 5087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:52.940675 5087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36337
I20260619 17:00:52.940730 5153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36337 every 8 connection(s)
I20260619 17:00:52.942267 5087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:52.946400 5154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:52.951891 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 5087
I20260619 17:00:52.952095 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:52.952638 16576 external_mini_cluster.cc:1428] Setting key e5408bdb868feafe5682d5b6e4ec52a0
I20260619 17:00:52.958588 5154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3: Bootstrap starting.
I20260619 17:00:52.961309 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:52.954614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47417 (local address 127.0.0.1:36337)
0619 17:00:52.955540 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:52.955553 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:52.955601 (+ 48us) server_negotiation.cc:407] Connection header received
0619 17:00:52.957519 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:52.957547 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:52.957953 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:52.958318 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:52.959595 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:52.959602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:52.959618 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:52.959643 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:52.959710 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:52.959732 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:52.959913 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:52.960028 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 17:00:52.960298 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":456,"threads_started":1}
I20260619 17:00:52.962319 5154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:52.963408 5154 log.cc:824] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:52.965622 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:00:52.966954 5154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3: No bootstrap required, opened a new log
I20260619 17:00:52.974087 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:52.971186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53839 (local address 127.0.0.1:36337)
0619 17:00:52.971400 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:52.971409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:52.971818 (+ 409us) server_negotiation.cc:407] Connection header received
0619 17:00:52.972068 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:52.972072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:52.972179 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:52.972313 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:52.973362 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:52.973367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:52.973370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:52.973387 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:52.973436 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:52.973458 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:52.973606 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:52.973703 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 17:00:52.973783 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 17:00:52.977424 5154 raft_consensus.cc:348] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } }
I20260619 17:00:52.977895 5154 raft_consensus.cc:374] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:52.978024 5154 raft_consensus.cc:729] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35e5dfdbda843d2a8c9ba3dc00b4bd3, State: Initialized, Role: FOLLOWER
I20260619 17:00:52.978649 5154 consensus_queue.cc:260] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } }
I20260619 17:00:52.978909 5154 raft_consensus.cc:388] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:52.979063 5154 raft_consensus.cc:482] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:52.979265 5154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:52.982023 5154 raft_consensus.cc:504] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } }
I20260619 17:00:52.982658 5154 leader_election.cc:302] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d35e5dfdbda843d2a8c9ba3dc00b4bd3; no voters:
I20260619 17:00:52.983352 5154 leader_election.cc:288] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:52.983566 5164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:52.984382 5164 raft_consensus.cc:686] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [term 1 LEADER]: Becoming Leader. State: Replica: d35e5dfdbda843d2a8c9ba3dc00b4bd3, State: Running, Role: LEADER
I20260619 17:00:52.985052 5164 consensus_queue.cc:237] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } }
I20260619 17:00:52.985718 5154 sys_catalog.cc:564] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:52.988662 5166 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d35e5dfdbda843d2a8c9ba3dc00b4bd3. Latest consensus state: current_term: 1 leader_uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } } }
I20260619 17:00:52.988551 5165 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35e5dfdbda843d2a8c9ba3dc00b4bd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } } }
I20260619 17:00:52.989218 5165 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:52.989218 5166 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:52.990303 5173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:52.997359 5173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:53.010509 5173 catalog_manager.cc:1347] Generated new cluster ID: 494a9bf6ed93409d90ec37d09411682f
I20260619 17:00:53.010679 5173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:53.028417 5173 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:53.028918 5173 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:53.037169 5173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d35e5dfdbda843d2a8c9ba3dc00b4bd3: Generated new TSK 0
I20260619 17:00:53.038398 5173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:53.053812 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5087
W20260619 17:00:53.055130 5149 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47417
W20260619 17:00:53.055984 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5087: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:00:53.056197 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 5087
2026-06-19T17:00:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:53 dist-test-slave-3ffd krb5kdc[5185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:53 dist-test-slave-3ffd krb5kdc[5185](info): set up 2 sockets
Jun 19 17:00:53 dist-test-slave-3ffd krb5kdc[5185](info): commencing operation
krb5kdc: starting...
W20260619 17:00:54.169034 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:54 dist-test-slave-3ffd krb5kdc[5185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:00:54.448557 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:54.807928 5198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:54.808373 5198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:54.808550 5198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:54.816308 5198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:54.816480 5198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:54.816583 5198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:54.816697 5198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:54.816789 5198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:54.827598 5198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:54.829205 5198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:54.830672 5198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:54.843897 5206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:54.843873 5204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:54.844290 5203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:54.844936 5198 server_base.cc:1034] running on GCE node
I20260619 17:00:54.845914 5198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:54.847551 5198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:54.848807 5198 hybrid_clock.cc:648] HybridClock initialized: now 1781888454848749 us; error 64 us; skew 500 ppm
I20260619 17:00:54.849206 5198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:54 dist-test-slave-3ffd krb5kdc[5185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:54.859669 5198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:00:54.861554 5198 webserver.cc:460] Webserver started at http://127.0.0.1:40999/ using document root <none> and password file <none>
I20260619 17:00:54.862382 5198 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:54.862560 5198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:54.862907 5198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:54.866279 5198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "5b31354402844f62886895eddd1a5ecb"
format_stamp: "Formatted at 2026-06-19 17:00:54 on dist-test-slave-3ffd"
server_key: "ea594da7baca2108259647c01dad9db5"
server_key_iv: "d9bdf974eb6f1a94f4d92c34251a000c"
server_key_version: "encryptionkey@0"
I20260619 17:00:54.867167 5198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "5b31354402844f62886895eddd1a5ecb"
format_stamp: "Formatted at 2026-06-19 17:00:54 on dist-test-slave-3ffd"
server_key: "ea594da7baca2108259647c01dad9db5"
server_key_iv: "d9bdf974eb6f1a94f4d92c34251a000c"
server_key_version: "encryptionkey@0"
I20260619 17:00:54.873095 5198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 17:00:54.877100 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:54.878638 5198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 17:00:54.878854 5198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "5b31354402844f62886895eddd1a5ecb"
format_stamp: "Formatted at 2026-06-19 17:00:54 on dist-test-slave-3ffd"
server_key: "ea594da7baca2108259647c01dad9db5"
server_key_iv: "d9bdf974eb6f1a94f4d92c34251a000c"
server_key_version: "encryptionkey@0"
I20260619 17:00:54.879084 5198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:54.890798 5198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:54.895714 5198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:54.896030 5198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:54.915825 5198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43271
I20260619 17:00:54.915843 5264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43271 every 8 connection(s)
I20260619 17:00:54.917434 5198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:54.921694 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:54.925900 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 5198
I20260619 17:00:54.926129 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:54.926556 16576 external_mini_cluster.cc:1428] Setting key c073678d90e00b220fbc6dea3787b79f
I20260619 17:00:54.935241 5265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb: Bootstrap starting.
I20260619 17:00:54.938740 5265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:54.940295 5265 log.cc:824] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb: Log is configured to *not* fsync() on all Append() calls
Jun 19 17:00:54 dist-test-slave-3ffd krb5kdc[5185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 17:00:54.943645 5265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb: No bootstrap required, opened a new log
I20260619 17:00:54.951748 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:54.928789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:41691 (local address 127.0.0.1:43271)
0619 17:00:54.929650 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:54.929664 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:54.929709 (+ 45us) server_negotiation.cc:407] Connection header received
0619 17:00:54.931441 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:54.931468 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:54.931989 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:54.932533 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:54.934115 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:54.935641 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:54.937130 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:54.937863 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:54.938097 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:54.943207 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:54.943216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:54.943232 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:54.943277 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:54.946753 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:54.947743 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:54.947751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:54.947758 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:54.947978 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:54.948557 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:54.948561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:54.948563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:54.949184 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:54.949559 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:54.950136 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 17:00:54.950498 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":361,"threads_started":1}
I20260619 17:00:54.952358 5265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31354402844f62886895eddd1a5ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } }
I20260619 17:00:54.952870 5265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:54.952976 5265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b31354402844f62886895eddd1a5ecb, State: Initialized, Role: FOLLOWER
I20260619 17:00:54.953696 5265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31354402844f62886895eddd1a5ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } }
I20260619 17:00:54.953962 5265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:54.954139 5265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:54.954327 5265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:54.956542 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:00:54.957265 5265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31354402844f62886895eddd1a5ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } }
I20260619 17:00:54.957814 5265 leader_election.cc:302] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b31354402844f62886895eddd1a5ecb; no voters:
I20260619 17:00:54.958519 5265 leader_election.cc:288] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:54.958635 5270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:54.960208 5270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [term 1 LEADER]: Becoming Leader. State: Replica: 5b31354402844f62886895eddd1a5ecb, State: Running, Role: LEADER
I20260619 17:00:54.960947 5270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31354402844f62886895eddd1a5ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } }
I20260619 17:00:54.961243 5265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:54.964601 5272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b31354402844f62886895eddd1a5ecb. Latest consensus state: current_term: 1 leader_uuid: "5b31354402844f62886895eddd1a5ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31354402844f62886895eddd1a5ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } } }
I20260619 17:00:54.965175 5272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:54.965930 5271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b31354402844f62886895eddd1a5ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31354402844f62886895eddd1a5ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } } }
I20260619 17:00:54.966313 5271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:54.969034 5281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:54.976609 5281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:55.002597 5281 catalog_manager.cc:1347] Generated new cluster ID: 3d05eaf0dc4443368429132bd9ccab35
I20260619 17:00:55.002729 5281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:55.005460 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:54.982540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:44145 (local address 127.0.0.1:43271)
0619 17:00:54.982978 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:54.982985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:54.983285 (+ 300us) server_negotiation.cc:407] Connection header received
0619 17:00:54.983559 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:54.983578 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:54.983691 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:54.983784 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:54.986125 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:54.987288 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:54.989927 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:54.990908 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:54.991067 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:54.995709 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:54.995722 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:54.995731 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:00:54.995799 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:55.000608 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:55.001885 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:55.001892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:55.001895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:55.002063 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:00:55.002734 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:00:55.002742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:00:55.002748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:00:55.003195 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:55.003484 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:55.004838 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0619 17:00:55.005068 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":282}
I20260619 17:00:55.013793 5293 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 17:00:55.030452 5281 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:55.031842 5281 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:55.046304 5281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b31354402844f62886895eddd1a5ecb: Generated new TSK 0
I20260619 17:00:55.047665 5281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:55.061853 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5198
I20260619 17:00:55.063198 5260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:00:55.158154 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 5198
W20260619 17:00:55.177069 5183 connection.cc:535] client connection to 127.0.0.1:43271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43271): Connection reset by peer (error 104)
2026-06-19T17:00:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2102 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:55 dist-test-slave-3ffd krb5kdc[5297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:55 dist-test-slave-3ffd krb5kdc[5297](info): set up 2 sockets
Jun 19 17:00:55 dist-test-slave-3ffd krb5kdc[5297](info): commencing operation
krb5kdc: starting...
W20260619 17:00:56.268958 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:56 dist-test-slave-3ffd krb5kdc[5297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:00:56.548681 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:56.903280 5310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:56.903635 5310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:56.903812 5310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:56.911442 5310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:56.911788 5310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:56.911922 5310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:56.912039 5310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:56.912125 5310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:56.922816 5310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:56.924321 5310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:56.925750 5310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:56.938766 5315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:56.938657 5316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:56.938717 5318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:56.939709 5310 server_base.cc:1034] running on GCE node
I20260619 17:00:56.940716 5310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:56.942399 5310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:56.943660 5310 hybrid_clock.cc:648] HybridClock initialized: now 1781888456943601 us; error 69 us; skew 500 ppm
I20260619 17:00:56.944044 5310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:56 dist-test-slave-3ffd krb5kdc[5297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:56.947631 5310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:00:56.949447 5310 webserver.cc:460] Webserver started at http://127.0.0.1:39229/ using document root <none> and password file <none>
I20260619 17:00:56.950232 5310 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:56.950393 5310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:56.950737 5310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:56.954144 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "31c4163f8eda4c7abc6ba4659028aca0"
format_stamp: "Formatted at 2026-06-19 17:00:56 on dist-test-slave-3ffd"
server_key: "f4574b30982374588fd18ad597ed01e4"
server_key_iv: "2d27645679ba0e2a5322297f6d2e6aff"
server_key_version: "encryptionkey@0"
I20260619 17:00:56.955039 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "31c4163f8eda4c7abc6ba4659028aca0"
format_stamp: "Formatted at 2026-06-19 17:00:56 on dist-test-slave-3ffd"
server_key: "f4574b30982374588fd18ad597ed01e4"
server_key_iv: "2d27645679ba0e2a5322297f6d2e6aff"
server_key_version: "encryptionkey@0"
I20260619 17:00:56.960740 5310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 17:00:56.964612 5325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:56.966158 5310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 17:00:56.966382 5310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "31c4163f8eda4c7abc6ba4659028aca0"
format_stamp: "Formatted at 2026-06-19 17:00:56 on dist-test-slave-3ffd"
server_key: "f4574b30982374588fd18ad597ed01e4"
server_key_iv: "2d27645679ba0e2a5322297f6d2e6aff"
server_key_version: "encryptionkey@0"
I20260619 17:00:56.966619 5310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:56.988233 5310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:56.993270 5310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:56.993610 5310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:57.013253 5310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46427
I20260619 17:00:57.013281 5376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46427 every 8 connection(s)
I20260619 17:00:57.014809 5310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:57.016563 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 5310
I20260619 17:00:57.016757 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:57.017208 16576 external_mini_cluster.cc:1428] Setting key de7d611ab2095e72a5fba0ffbdc72bce
I20260619 17:00:57.019879 5377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:57.033756 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:57.019215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:37531 (local address 127.0.0.1:46427)
0619 17:00:57.020042 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:57.020058 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:57.020110 (+ 52us) server_negotiation.cc:407] Connection header received
0619 17:00:57.022653 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:57.022684 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:57.023280 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:57.023786 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:57.024373 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:57.026469 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:57.027839 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:57.028940 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:57.029143 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:57.030554 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:57.030565 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:57.030586 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:57.031408 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:57.031523 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:57.031566 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:57.031755 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:57.032032 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 17:00:57.032280 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":339,"threads_started":1}
I20260619 17:00:57.035171 5377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0: Bootstrap starting.
I20260619 17:00:57.038571 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:00:57.039184 5377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:57.040573 5377 log.cc:824] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:57.043927 5377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0: No bootstrap required, opened a new log
I20260619 17:00:57.053193 5377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4163f8eda4c7abc6ba4659028aca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } }
I20260619 17:00:57.053843 5377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:57.053988 5377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c4163f8eda4c7abc6ba4659028aca0, State: Initialized, Role: FOLLOWER
I20260619 17:00:57.054739 5377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4163f8eda4c7abc6ba4659028aca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } }
I20260619 17:00:57.055040 5377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:57.055199 5377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:57.055405 5377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:57.055976 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:57.044079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:40833 (local address 127.0.0.1:46427)
0619 17:00:57.044294 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:57.044300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:57.048878 (+ 4578us) server_negotiation.cc:407] Connection header received
0619 17:00:57.049035 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:57.049041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:57.049144 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:57.049277 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:57.049838 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:57.051048 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:57.052837 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:00:57.053544 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:00:57.053704 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:00:57.055070 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:57.055077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:57.055083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:57.055110 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:57.055162 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:57.055194 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:57.055326 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:57.055558 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 17:00:57.055625 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 17:00:57.058106 5377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4163f8eda4c7abc6ba4659028aca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } }
I20260619 17:00:57.058600 5377 leader_election.cc:302] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31c4163f8eda4c7abc6ba4659028aca0; no voters:
I20260619 17:00:57.059233 5377 leader_election.cc:288] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:57.059412 5387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:57.060420 5387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [term 1 LEADER]: Becoming Leader. State: Replica: 31c4163f8eda4c7abc6ba4659028aca0, State: Running, Role: LEADER
I20260619 17:00:57.061025 5387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4163f8eda4c7abc6ba4659028aca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } }
I20260619 17:00:57.061690 5377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:57.064230 5389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31c4163f8eda4c7abc6ba4659028aca0. Latest consensus state: current_term: 1 leader_uuid: "31c4163f8eda4c7abc6ba4659028aca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4163f8eda4c7abc6ba4659028aca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } } }
I20260619 17:00:57.064667 5389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:57.065459 5388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31c4163f8eda4c7abc6ba4659028aca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4163f8eda4c7abc6ba4659028aca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } } }
I20260619 17:00:57.065857 5388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:57.066630 5394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:57.073622 5394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:57.084880 5394 catalog_manager.cc:1347] Generated new cluster ID: 287230ac76464f2f92527209b6580075
I20260619 17:00:57.085031 5394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:57.099426 5394 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:57.100994 5394 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:57.108795 5394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31c4163f8eda4c7abc6ba4659028aca0: Generated new TSK 0
I20260619 17:00:57.109746 5394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:57.121455 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5310
W20260619 17:00:57.123606 5372 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.146:37531
W20260619 17:00:57.124467 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5310: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:00:57.124632 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 5310
W20260619 17:00:57.140479 5295 connection.cc:535] client connection to 127.0.0.1:46427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46427): Connection reset by peer (error 104)
2026-06-19T17:00:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:57 dist-test-slave-3ffd krb5kdc[5409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:57 dist-test-slave-3ffd krb5kdc[5409](info): set up 2 sockets
Jun 19 17:00:57 dist-test-slave-3ffd krb5kdc[5409](info): commencing operation
krb5kdc: starting...
W20260619 17:00:58.252969 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:00:58 dist-test-slave-3ffd krb5kdc[5409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:00:58.531858 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:00:58.891739 5422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:00:58.892072 5422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:00:58.892264 5422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:00:58.899853 5422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:00:58.899987 5422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:00:58.900034 5422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:00:58.900096 5422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:00:58.900207 5422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:00:58.911334 5422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:00:58.912739 5422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:00:58.914063 5422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:00:58.926848 5428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:58.926898 5430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:00:58.926815 5427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:00:58.927870 5422 server_base.cc:1034] running on GCE node
I20260619 17:00:58.928822 5422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:00:58.930384 5422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:00:58.931643 5422 hybrid_clock.cc:648] HybridClock initialized: now 1781888458931594 us; error 47 us; skew 500 ppm
I20260619 17:00:58.932021 5422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:00:58 dist-test-slave-3ffd krb5kdc[5409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:00:58.935518 5422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:00:58.937351 5422 webserver.cc:460] Webserver started at http://127.0.0.1:36219/ using document root <none> and password file <none>
I20260619 17:00:58.938210 5422 fs_manager.cc:359] Metadata directory not provided
I20260619 17:00:58.938369 5422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:00:58.938740 5422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:00:58.942083 5422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "28bd81f9ee4d423e960cbc5602623255"
format_stamp: "Formatted at 2026-06-19 17:00:58 on dist-test-slave-3ffd"
server_key: "58c6d6f2807d2153f96512ea68cafef4"
server_key_iv: "70af53fb8d8e83b5bd2a56d9f5db4cf6"
server_key_version: "encryptionkey@0"
I20260619 17:00:58.942946 5422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "28bd81f9ee4d423e960cbc5602623255"
format_stamp: "Formatted at 2026-06-19 17:00:58 on dist-test-slave-3ffd"
server_key: "58c6d6f2807d2153f96512ea68cafef4"
server_key_iv: "70af53fb8d8e83b5bd2a56d9f5db4cf6"
server_key_version: "encryptionkey@0"
I20260619 17:00:58.948887 5422 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 17:00:58.952988 5437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:00:58.954500 5422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 17:00:58.954718 5422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "28bd81f9ee4d423e960cbc5602623255"
format_stamp: "Formatted at 2026-06-19 17:00:58 on dist-test-slave-3ffd"
server_key: "58c6d6f2807d2153f96512ea68cafef4"
server_key_iv: "70af53fb8d8e83b5bd2a56d9f5db4cf6"
server_key_version: "encryptionkey@0"
I20260619 17:00:58.954952 5422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:00:58.961516 5422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:00:58.966293 5422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:00:58.966651 5422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:00:58.986691 5422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46605
I20260619 17:00:58.986747 5488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46605 every 8 connection(s)
I20260619 17:00:58.988226 5422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:00:58.989663 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 5422
I20260619 17:00:58.989845 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:00:58.990381 16576 external_mini_cluster.cc:1428] Setting key 72ecfcd8aa570b79d34f38c042e0d4de
I20260619 17:00:58.993360 5489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:00:58.998826 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:58.992375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:59815 (local address 127.0.0.1:46605)
0619 17:00:58.993237 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:58.993249 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:58.993293 (+ 44us) server_negotiation.cc:407] Connection header received
0619 17:00:58.995063 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:58.995089 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:58.995449 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:58.995892 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:58.997044 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:58.997050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:58.997065 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:58.997080 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:58.997168 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:58.997192 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:58.997447 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:58.997611 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 17:00:58.997812 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":363,"threads_started":1}
I20260619 17:00:59.003059 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:00:59.005687 5489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255: Bootstrap starting.
I20260619 17:00:59.009815 5489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255: Neither blocks nor log segments found. Creating new log.
I20260619 17:00:59.010535 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:00:59.007703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:35263 (local address 127.0.0.1:46605)
0619 17:00:59.007963 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 17:00:59.007972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:00:59.008386 (+ 414us) server_negotiation.cc:407] Connection header received
0619 17:00:59.008611 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:00:59.008616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:00:59.008669 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:00:59.008774 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:00:59.009765 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:00:59.009770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:00:59.009773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:00:59.009789 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:00:59.009833 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:00:59.009849 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:00:59.009975 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 17:00:59.010146 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 17:00:59.010204 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 17:00:59.011336 5489 log.cc:824] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255: Log is configured to *not* fsync() on all Append() calls
I20260619 17:00:59.014941 5489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255: No bootstrap required, opened a new log
I20260619 17:00:59.023267 5489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bd81f9ee4d423e960cbc5602623255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260619 17:00:59.023905 5489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:00:59.024032 5489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bd81f9ee4d423e960cbc5602623255, State: Initialized, Role: FOLLOWER
I20260619 17:00:59.024829 5489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bd81f9ee4d423e960cbc5602623255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260619 17:00:59.025086 5489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:00:59.025279 5489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:00:59.025462 5489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:00:59.027805 5489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bd81f9ee4d423e960cbc5602623255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260619 17:00:59.028446 5489 leader_election.cc:302] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28bd81f9ee4d423e960cbc5602623255; no voters:
I20260619 17:00:59.029119 5489 leader_election.cc:288] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:00:59.029296 5499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:00:59.030648 5499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [term 1 LEADER]: Becoming Leader. State: Replica: 28bd81f9ee4d423e960cbc5602623255, State: Running, Role: LEADER
I20260619 17:00:59.031333 5499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bd81f9ee4d423e960cbc5602623255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260619 17:00:59.031724 5489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:00:59.034824 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28bd81f9ee4d423e960cbc5602623255. Latest consensus state: current_term: 1 leader_uuid: "28bd81f9ee4d423e960cbc5602623255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bd81f9ee4d423e960cbc5602623255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } } }
I20260619 17:00:59.034895 5500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28bd81f9ee4d423e960cbc5602623255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bd81f9ee4d423e960cbc5602623255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } } }
I20260619 17:00:59.035398 5500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:59.035665 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255 [sys.catalog]: This master's current role is: LEADER
I20260619 17:00:59.037333 5505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:00:59.044647 5505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:00:59.055366 5505 catalog_manager.cc:1347] Generated new cluster ID: aeb344b9b4c64080a747022500dd5276
I20260619 17:00:59.055518 5505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:00:59.066349 5505 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:00:59.066813 5505 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:00:59.073454 5505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28bd81f9ee4d423e960cbc5602623255: Generated new TSK 0
I20260619 17:00:59.074384 5505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:00:59.088336 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5422
W20260619 17:00:59.089768 5484 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.146:59815
W20260619 17:00:59.090626 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5422: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:00:59.090798 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 5422
2026-06-19T17:00:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:00:59 dist-test-slave-3ffd krb5kdc[5520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:00:59 dist-test-slave-3ffd krb5kdc[5520](info): set up 2 sockets
Jun 19 17:00:59 dist-test-slave-3ffd krb5kdc[5520](info): commencing operation
krb5kdc: starting...
W20260619 17:01:00.200887 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:00 dist-test-slave-3ffd krb5kdc[5520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:01:00.485873 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:00.840045 5533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:00.840579 5533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:00.840767 5533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:00.848667 5533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:00.848803 5533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:00.848888 5533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:00.849007 5533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:00.849092 5533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:00.859726 5533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:00.861208 5533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:00.862637 5533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:00.876171 5539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:00.876189 5541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:00.876525 5538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:00.877257 5533 server_base.cc:1034] running on GCE node
I20260619 17:01:00.878216 5533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:00.879814 5533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:00.881089 5533 hybrid_clock.cc:648] HybridClock initialized: now 1781888460880992 us; error 84 us; skew 500 ppm
I20260619 17:01:00.881492 5533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:00 dist-test-slave-3ffd krb5kdc[5520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:00.889987 5533 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:01:00.891644 5533 webserver.cc:460] Webserver started at http://127.0.0.1:38099/ using document root <none> and password file <none>
I20260619 17:01:00.892447 5533 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:00.892563 5533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:00.892916 5533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:00.896106 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "b32be1a630fe4bd99abd34c3eaa7bf32"
format_stamp: "Formatted at 2026-06-19 17:01:00 on dist-test-slave-3ffd"
server_key: "2a9a02611a4788efd97a2237e35211d9"
server_key_iv: "5260b7ac9f6f8156c787baecf0bc1031"
server_key_version: "encryptionkey@0"
I20260619 17:01:00.896996 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "b32be1a630fe4bd99abd34c3eaa7bf32"
format_stamp: "Formatted at 2026-06-19 17:01:00 on dist-test-slave-3ffd"
server_key: "2a9a02611a4788efd97a2237e35211d9"
server_key_iv: "5260b7ac9f6f8156c787baecf0bc1031"
server_key_version: "encryptionkey@0"
I20260619 17:01:00.902590 5533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 17:01:00.906441 5548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:00.907928 5533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 17:01:00.908192 5533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "b32be1a630fe4bd99abd34c3eaa7bf32"
format_stamp: "Formatted at 2026-06-19 17:01:00 on dist-test-slave-3ffd"
server_key: "2a9a02611a4788efd97a2237e35211d9"
server_key_iv: "5260b7ac9f6f8156c787baecf0bc1031"
server_key_version: "encryptionkey@0"
I20260619 17:01:00.908418 5533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:00.923843 5533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:00.928321 5533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:00.928627 5533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:00.948923 5533 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42595
I20260619 17:01:00.948959 5599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42595 every 8 connection(s)
I20260619 17:01:00.950503 5533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:00.954357 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 5533
I20260619 17:01:00.954512 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:00.955057 16576 external_mini_cluster.cc:1428] Setting key 00b0284b306da2c5f350081dc9783bf3
I20260619 17:01:00.955184 5600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:00.972003 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32: Bootstrap starting.
Jun 19 17:01:00 dist-test-slave-3ffd krb5kdc[5520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 17:01:00.976915 5600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:00.978543 5600 log.cc:824] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:00.982162 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32: No bootstrap required, opened a new log
I20260619 17:01:00.983490 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:00.957068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:53341 (local address 127.0.0.1:42595)
0619 17:01:00.957929 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:00.957948 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:00.958004 (+ 56us) server_negotiation.cc:407] Connection header received
0619 17:01:00.960751 (+ 2747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:00.960786 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:00.961446 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:00.962011 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:00.963537 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:00.965845 (+ 2308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:00.967345 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:00.968679 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:00.968914 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:00.973554 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:00.973568 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:00.973588 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:00.973651 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:00.978377 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:00.979559 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:00.979567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:00.979574 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:00.979795 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:00.980432 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:00.980439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:00.980447 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:00.981014 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:00.981395 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:00.981978 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 17:01:00.982403 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":350,"threads_started":1}
I20260619 17:01:00.988039 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:00.991918 5600 raft_consensus.cc:348] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32be1a630fe4bd99abd34c3eaa7bf32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } }
I20260619 17:01:00.992568 5600 raft_consensus.cc:374] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:01:00.992724 5600 raft_consensus.cc:729] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32be1a630fe4bd99abd34c3eaa7bf32, State: Initialized, Role: FOLLOWER
I20260619 17:01:00.993570 5600 consensus_queue.cc:260] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32be1a630fe4bd99abd34c3eaa7bf32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } }
I20260619 17:01:00.993885 5600 raft_consensus.cc:388] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:01:00.994062 5600 raft_consensus.cc:482] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:01:00.994258 5600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:00.996924 5600 raft_consensus.cc:504] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32be1a630fe4bd99abd34c3eaa7bf32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } }
I20260619 17:01:00.997539 5600 leader_election.cc:302] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b32be1a630fe4bd99abd34c3eaa7bf32; no voters:
I20260619 17:01:00.998363 5600 leader_election.cc:288] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:01:00.998526 5610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:01:01.000419 5610 raft_consensus.cc:686] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [term 1 LEADER]: Becoming Leader. State: Replica: b32be1a630fe4bd99abd34c3eaa7bf32, State: Running, Role: LEADER
I20260619 17:01:01.001204 5610 consensus_queue.cc:237] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32be1a630fe4bd99abd34c3eaa7bf32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } }
I20260619 17:01:01.001416 5600 sys_catalog.cc:564] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:01.007298 5612 sys_catalog.cc:455] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b32be1a630fe4bd99abd34c3eaa7bf32. Latest consensus state: current_term: 1 leader_uuid: "b32be1a630fe4bd99abd34c3eaa7bf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32be1a630fe4bd99abd34c3eaa7bf32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } } }
I20260619 17:01:01.008100 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b32be1a630fe4bd99abd34c3eaa7bf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32be1a630fe4bd99abd34c3eaa7bf32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } } }
I20260619 17:01:01.008502 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:01.009227 5612 sys_catalog.cc:458] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:01.011060 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:00.992810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:35305 (local address 127.0.0.1:42595)
0619 17:01:00.993008 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:00.993013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:00.993527 (+ 514us) server_negotiation.cc:407] Connection header received
0619 17:01:00.993750 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:00.993755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:00.993859 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:00.993962 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:00.995533 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:00.996406 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:00.997780 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:00.998303 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:00.998464 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:01.001543 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:01.001551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:01.001559 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:01.001614 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:01.005120 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:01.006383 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:01.006393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:01.006399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:01.006633 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:01.007482 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:01.007491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:01.007501 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:01.008189 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:01.008641 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:01.010223 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0619 17:01:01.010499 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 17:01:01.016875 5619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:01.023034 5619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:01.032900 5619 catalog_manager.cc:1347] Generated new cluster ID: b43a1337632a4466a1f6feafae228a22
I20260619 17:01:01.033046 5619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:01.049249 5619 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:01.051254 5619 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:01.066807 5619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b32be1a630fe4bd99abd34c3eaa7bf32: Generated new TSK 0
I20260619 17:01:01.068024 5619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:01.078423 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5533
I20260619 17:01:01.079938 5595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:01:01.175279 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 5533
W20260619 17:01:01.193343 5518 connection.cc:535] client connection to 127.0.0.1:42595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42595): Connection reset by peer (error 104)
2026-06-19T17:01:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2089 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:01 dist-test-slave-3ffd krb5kdc[5633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:01 dist-test-slave-3ffd krb5kdc[5633](info): set up 2 sockets
Jun 19 17:01:01 dist-test-slave-3ffd krb5kdc[5633](info): commencing operation
krb5kdc: starting...
W20260619 17:01:02.309010 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:02 dist-test-slave-3ffd krb5kdc[5633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:01:02.597491 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:02.961267 5646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:02.961679 5646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:02.961845 5646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:02.970129 5646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:02.970327 5646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:02.970434 5646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:02.970546 5646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:02.970629 5646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:02.981295 5646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:02.982784 5646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:02.984303 5646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:02.997434 5652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:02.997382 5651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:02.997702 5654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:02.998713 5646 server_base.cc:1034] running on GCE node
I20260619 17:01:02.999662 5646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:03.001472 5646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:03.002732 5646 hybrid_clock.cc:648] HybridClock initialized: now 1781888463002682 us; error 43 us; skew 500 ppm
I20260619 17:01:03.003183 5646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:03 dist-test-slave-3ffd krb5kdc[5633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:03.006920 5646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:01:03.008795 5646 webserver.cc:460] Webserver started at http://127.0.0.1:41607/ using document root <none> and password file <none>
I20260619 17:01:03.009618 5646 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:03.009765 5646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:03.010120 5646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:03.013563 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "f96a44adf79b4d198b93d2ce9f06f247"
format_stamp: "Formatted at 2026-06-19 17:01:03 on dist-test-slave-3ffd"
server_key: "554ea4a6a8ded5f2dccc7661b27887b9"
server_key_iv: "6d3a16c2e47c8fc7aa15cc2d5181ddb0"
server_key_version: "encryptionkey@0"
I20260619 17:01:03.014449 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "f96a44adf79b4d198b93d2ce9f06f247"
format_stamp: "Formatted at 2026-06-19 17:01:03 on dist-test-slave-3ffd"
server_key: "554ea4a6a8ded5f2dccc7661b27887b9"
server_key_iv: "6d3a16c2e47c8fc7aa15cc2d5181ddb0"
server_key_version: "encryptionkey@0"
I20260619 17:01:03.020283 5646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 17:01:03.024080 5661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:03.025759 5646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 17:01:03.025995 5646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "f96a44adf79b4d198b93d2ce9f06f247"
format_stamp: "Formatted at 2026-06-19 17:01:03 on dist-test-slave-3ffd"
server_key: "554ea4a6a8ded5f2dccc7661b27887b9"
server_key_iv: "6d3a16c2e47c8fc7aa15cc2d5181ddb0"
server_key_version: "encryptionkey@0"
I20260619 17:01:03.026221 5646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:03.051796 5646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:03.056838 5646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:03.057149 5646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:03.077558 5646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43019
I20260619 17:01:03.077613 5712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43019 every 8 connection(s)
I20260619 17:01:03.079058 5646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:03.083289 5713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:03.085412 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 5646
I20260619 17:01:03.085574 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:03.086059 16576 external_mini_cluster.cc:1428] Setting key 7f648e8c82f4ffd8f6e65c4b9852ad93
I20260619 17:01:03.099507 5713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247: Bootstrap starting.
I20260619 17:01:03.103178 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:03.088457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:55713 (local address 127.0.0.1:43019)
0619 17:01:03.089463 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:03.089481 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:03.089550 (+ 69us) server_negotiation.cc:407] Connection header received
0619 17:01:03.091493 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:03.091530 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:03.092239 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:03.092748 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:03.093326 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:03.095320 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:03.097076 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:03.098253 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:03.098462 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:03.099947 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:03.099958 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:03.099979 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:03.100814 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:03.100931 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:03.100973 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:03.101171 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:03.101531 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 17:01:03.101763 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":415,"threads_started":1}
I20260619 17:01:03.103952 5713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:03.105168 5713 log.cc:824] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:03.107916 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:03.108737 5713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247: No bootstrap required, opened a new log
I20260619 17:01:03.118654 5713 raft_consensus.cc:348] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a44adf79b4d198b93d2ce9f06f247" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } }
I20260619 17:01:03.119216 5713 raft_consensus.cc:374] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:01:03.119354 5713 raft_consensus.cc:729] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96a44adf79b4d198b93d2ce9f06f247, State: Initialized, Role: FOLLOWER
I20260619 17:01:03.120191 5713 consensus_queue.cc:260] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a44adf79b4d198b93d2ce9f06f247" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } }
I20260619 17:01:03.120436 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:03.113027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:46489 (local address 127.0.0.1:43019)
0619 17:01:03.113263 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:03.113272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:03.113550 (+ 278us) server_negotiation.cc:407] Connection header received
0619 17:01:03.113752 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:03.113756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:03.113832 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:03.113945 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:03.114621 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:03.115411 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:03.117476 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:03.117924 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:03.118042 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:03.119470 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:03.119480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:03.119485 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:03.119521 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:03.119564 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:03.119582 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:03.119677 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:03.119939 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 17:01:03.120013 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 17:01:03.120509 5713 raft_consensus.cc:388] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:01:03.120675 5713 raft_consensus.cc:482] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:01:03.120918 5713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:03.123535 5713 raft_consensus.cc:504] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a44adf79b4d198b93d2ce9f06f247" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } }
I20260619 17:01:03.124221 5713 leader_election.cc:302] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f96a44adf79b4d198b93d2ce9f06f247; no voters:
I20260619 17:01:03.124889 5713 leader_election.cc:288] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:01:03.125118 5723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:01:03.126461 5723 raft_consensus.cc:686] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [term 1 LEADER]: Becoming Leader. State: Replica: f96a44adf79b4d198b93d2ce9f06f247, State: Running, Role: LEADER
I20260619 17:01:03.127082 5723 consensus_queue.cc:237] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a44adf79b4d198b93d2ce9f06f247" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } }
I20260619 17:01:03.127269 5713 sys_catalog.cc:564] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:03.131129 5724 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f96a44adf79b4d198b93d2ce9f06f247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a44adf79b4d198b93d2ce9f06f247" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } } }
I20260619 17:01:03.131587 5724 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:03.132180 5725 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f96a44adf79b4d198b93d2ce9f06f247. Latest consensus state: current_term: 1 leader_uuid: "f96a44adf79b4d198b93d2ce9f06f247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a44adf79b4d198b93d2ce9f06f247" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } } }
I20260619 17:01:03.132571 5725 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:03.132845 5732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:03.140379 5732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:03.153731 5732 catalog_manager.cc:1347] Generated new cluster ID: f1cac64a5d3b412c9022a131dbcad9f0
I20260619 17:01:03.153936 5732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:03.166184 5732 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:03.168237 5732 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:03.194351 5732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f96a44adf79b4d198b93d2ce9f06f247: Generated new TSK 0
I20260619 17:01:03.195621 5732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:03.211898 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5646
W20260619 17:01:03.214366 5708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.146:55713
W20260619 17:01:03.215193 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5646: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:01:03.215349 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 5646
W20260619 17:01:03.233371 5631 connection.cc:535] client connection to 127.0.0.1:43019 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43019): Connection reset by peer (error 104)
2026-06-19T17:01:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2035 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:03 dist-test-slave-3ffd krb5kdc[5744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:03 dist-test-slave-3ffd krb5kdc[5744](info): set up 2 sockets
Jun 19 17:01:03 dist-test-slave-3ffd krb5kdc[5744](info): commencing operation
krb5kdc: starting...
W20260619 17:01:04.348963 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:04 dist-test-slave-3ffd krb5kdc[5744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 17:01:04.632534 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:04.991225 5757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:04.991564 5757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:04.991722 5757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:04.999419 5757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:04.999668 5757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:04.999779 5757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:04.999887 5757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:04.999958 5757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:05.010519 5757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:05.011932 5757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:05.013331 5757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:05.026481 5762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:05.026644 5765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:05.026237 5763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:05.027639 5757 server_base.cc:1034] running on GCE node
I20260619 17:01:05.028585 5757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:05.030040 5757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:05.031340 5757 hybrid_clock.cc:648] HybridClock initialized: now 1781888465031302 us; error 33 us; skew 500 ppm
I20260619 17:01:05.031710 5757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:05 dist-test-slave-3ffd krb5kdc[5744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:05.039712 5757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 17:01:05.041617 5757 webserver.cc:460] Webserver started at http://127.0.0.1:40819/ using document root <none> and password file <none>
I20260619 17:01:05.042403 5757 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:05.042560 5757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:05.042917 5757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:05.046268 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "0cff1672c4e94b8c8d03c2f8097aa55a"
format_stamp: "Formatted at 2026-06-19 17:01:05 on dist-test-slave-3ffd"
server_key: "da562908f2e5d4b548aef358e272d4ed"
server_key_iv: "200323422cf91cb88f0c8adaa89b265c"
server_key_version: "encryptionkey@0"
I20260619 17:01:05.047099 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "0cff1672c4e94b8c8d03c2f8097aa55a"
format_stamp: "Formatted at 2026-06-19 17:01:05 on dist-test-slave-3ffd"
server_key: "da562908f2e5d4b548aef358e272d4ed"
server_key_iv: "200323422cf91cb88f0c8adaa89b265c"
server_key_version: "encryptionkey@0"
I20260619 17:01:05.052942 5757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 17:01:05.057104 5772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:05.058599 5757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 17:01:05.058815 5757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "0cff1672c4e94b8c8d03c2f8097aa55a"
format_stamp: "Formatted at 2026-06-19 17:01:05 on dist-test-slave-3ffd"
server_key: "da562908f2e5d4b548aef358e272d4ed"
server_key_iv: "200323422cf91cb88f0c8adaa89b265c"
server_key_version: "encryptionkey@0"
I20260619 17:01:05.059013 5757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:05.066969 5757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:05.074700 5757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:05.075387 5757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:05.099298 5757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41823
I20260619 17:01:05.099344 5823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41823 every 8 connection(s)
I20260619 17:01:05.101481 5757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:05.101764 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 5757
I20260619 17:01:05.101946 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:05.102597 16576 external_mini_cluster.cc:1428] Setting key f07c0322d8cffe9f6284d972c858fec7
I20260619 17:01:05.107472 5824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:05.111613 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:05.104693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:43759 (local address 127.0.0.1:41823)
0619 17:01:05.105623 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:05.105636 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:05.105680 (+ 44us) server_negotiation.cc:407] Connection header received
0619 17:01:05.107369 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:05.107392 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:05.107849 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:05.108329 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:05.109414 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:05.109424 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:05.109445 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:05.109464 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:05.109564 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:05.109593 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:05.109849 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:05.110015 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 17:01:05.110238 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":404,"threads_started":1}
I20260619 17:01:05.116894 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:05.124639 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:05.121524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:55559 (local address 127.0.0.1:41823)
0619 17:01:05.121770 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:05.121780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:05.122255 (+ 475us) server_negotiation.cc:407] Connection header received
0619 17:01:05.122423 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:05.122427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:05.122483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:05.122636 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:05.123776 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:05.123782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:05.123786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:05.123806 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:05.123860 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:05.123881 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:05.124040 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:05.124186 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 17:01:05.124270 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 17:01:05.125761 5824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a: Bootstrap starting.
I20260619 17:01:05.129307 5824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:05.130493 5824 log.cc:824] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:05.133898 5824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a: No bootstrap required, opened a new log
I20260619 17:01:05.141795 5824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cff1672c4e94b8c8d03c2f8097aa55a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260619 17:01:05.142302 5824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:01:05.142436 5824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cff1672c4e94b8c8d03c2f8097aa55a, State: Initialized, Role: FOLLOWER
I20260619 17:01:05.143204 5824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cff1672c4e94b8c8d03c2f8097aa55a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260619 17:01:05.143546 5824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:01:05.143707 5824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:01:05.143913 5824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:05.146309 5824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cff1672c4e94b8c8d03c2f8097aa55a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260619 17:01:05.146905 5824 leader_election.cc:302] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cff1672c4e94b8c8d03c2f8097aa55a; no voters:
I20260619 17:01:05.147651 5824 leader_election.cc:288] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:01:05.147853 5834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:01:05.148638 5834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [term 1 LEADER]: Becoming Leader. State: Replica: 0cff1672c4e94b8c8d03c2f8097aa55a, State: Running, Role: LEADER
I20260619 17:01:05.149248 5834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cff1672c4e94b8c8d03c2f8097aa55a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260619 17:01:05.150080 5824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:05.152990 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cff1672c4e94b8c8d03c2f8097aa55a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cff1672c4e94b8c8d03c2f8097aa55a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } } }
I20260619 17:01:05.153268 5836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cff1672c4e94b8c8d03c2f8097aa55a. Latest consensus state: current_term: 1 leader_uuid: "0cff1672c4e94b8c8d03c2f8097aa55a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cff1672c4e94b8c8d03c2f8097aa55a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } } }
I20260619 17:01:05.153573 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:05.153602 5836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:05.155339 5842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:05.162503 5842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:05.173273 5842 catalog_manager.cc:1347] Generated new cluster ID: e3528969915c4888906721523b744db4
I20260619 17:01:05.173434 5842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:05.212797 5842 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:05.213335 5842 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:05.231349 5842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cff1672c4e94b8c8d03c2f8097aa55a: Generated new TSK 0
I20260619 17:01:05.232609 5842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:05.249135 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5757
W20260619 17:01:05.250404 5819 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.146:43759
W20260619 17:01:05.251173 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5757: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:01:05.251423 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 5757
2026-06-19T17:01:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2032 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:05 dist-test-slave-3ffd krb5kdc[5855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:05 dist-test-slave-3ffd krb5kdc[5855](info): set up 2 sockets
Jun 19 17:01:05 dist-test-slave-3ffd krb5kdc[5855](info): commencing operation
krb5kdc: starting...
W20260619 17:01:06.372920 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:06 dist-test-slave-3ffd krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:01:06.650803 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:33593
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:33593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:07.005332 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:07.005666 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:07.005867 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:07.013623 5868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:07.013762 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:07.013860 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:07.013976 5868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:07.014037 5868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:07.024569 5868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:33593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:07.026047 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:07.027484 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:07.040657 5876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:07.040726 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:07.041356 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:07.041803 5868 server_base.cc:1034] running on GCE node
I20260619 17:01:07.042676 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:07.044370 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:07.045629 5868 hybrid_clock.cc:648] HybridClock initialized: now 1781888467045586 us; error 63 us; skew 500 ppm
I20260619 17:01:07.046020 5868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:07 dist-test-slave-3ffd krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:07.054965 5868 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:01:07.056928 5868 webserver.cc:460] Webserver started at http://127.16.48.62:39127/ using document root <none> and password file <none>
I20260619 17:01:07.057713 5868 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:07.057876 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:07.058224 5868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:07.061652 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "9e01f8507e11425a830b28888476fa88"
format_stamp: "Formatted at 2026-06-19 17:01:07 on dist-test-slave-3ffd"
server_key: "5aa61364c6a61a678e09956b8907f5c1"
server_key_iv: "bb942d9be9a0937f8be41f20a4a84c23"
server_key_version: "encryptionkey@0"
I20260619 17:01:07.062529 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "9e01f8507e11425a830b28888476fa88"
format_stamp: "Formatted at 2026-06-19 17:01:07 on dist-test-slave-3ffd"
server_key: "5aa61364c6a61a678e09956b8907f5c1"
server_key_iv: "bb942d9be9a0937f8be41f20a4a84c23"
server_key_version: "encryptionkey@0"
I20260619 17:01:07.068854 5868 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 17:01:07.072865 5883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:07.074317 5868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 17:01:07.074537 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "9e01f8507e11425a830b28888476fa88"
format_stamp: "Formatted at 2026-06-19 17:01:07 on dist-test-slave-3ffd"
server_key: "5aa61364c6a61a678e09956b8907f5c1"
server_key_iv: "bb942d9be9a0937f8be41f20a4a84c23"
server_key_version: "encryptionkey@0"
I20260619 17:01:07.074781 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:07.091755 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:07.096290 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:07.096606 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:07.115969 5868 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:33593
I20260619 17:01:07.116010 5934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:33593 every 8 connection(s)
I20260619 17:01:07.117439 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:07.118429 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 5868
I20260619 17:01:07.118607 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:07.119179 16576 external_mini_cluster.cc:1428] Setting key 708c394eec8c304da423bf41a32ddfeb
I20260619 17:01:07.122328 5935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:07.136020 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88: Bootstrap starting.
Jun 19 17:01:07 dist-test-slave-3ffd krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:01:07.139729 5935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:07.141111 5935 log.cc:824] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:07.144392 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88: No bootstrap required, opened a new log
I20260619 17:01:07.146682 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:07.121318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59483 (local address 127.16.48.62:33593)
0619 17:01:07.122301 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:07.122324 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:07.124826 (+ 2502us) server_negotiation.cc:407] Connection header received
0619 17:01:07.127488 (+ 2662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:07.127516 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:07.128013 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:07.128556 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:07.130173 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:07.131515 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:07.133033 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:07.133849 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:07.134087 (+ 238us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:07.138638 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:07.138646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:07.138663 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:07.138710 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:07.142137 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:07.143054 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:07.143061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:07.143068 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:07.143238 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:07.143751 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:07.143756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:07.143758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:07.144370 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:07.144702 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:07.145232 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 17:01:07.145578 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":437,"threads_started":1}
I20260619 17:01:07.151127 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:07.154464 5935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e01f8507e11425a830b28888476fa88" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33593 } }
I20260619 17:01:07.155110 5935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:01:07.155232 5935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e01f8507e11425a830b28888476fa88, State: Initialized, Role: FOLLOWER
I20260619 17:01:07.156010 5935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e01f8507e11425a830b28888476fa88" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33593 } }
I20260619 17:01:07.156455 5935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:01:07.156638 5935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:01:07.156854 5935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:07.160559 5935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e01f8507e11425a830b28888476fa88" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33593 } }
I20260619 17:01:07.161216 5935 leader_election.cc:302] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e01f8507e11425a830b28888476fa88; no voters:
I20260619 17:01:07.161955 5935 leader_election.cc:288] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:01:07.162106 5945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:01:07.163296 5945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [term 1 LEADER]: Becoming Leader. State: Replica: 9e01f8507e11425a830b28888476fa88, State: Running, Role: LEADER
I20260619 17:01:07.164417 5945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e01f8507e11425a830b28888476fa88" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33593 } }
I20260619 17:01:07.164682 5935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:07.174326 5947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e01f8507e11425a830b28888476fa88. Latest consensus state: current_term: 1 leader_uuid: "9e01f8507e11425a830b28888476fa88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e01f8507e11425a830b28888476fa88" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33593 } } }
I20260619 17:01:07.174772 5947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:07.175338 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:07.156986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38531 (local address 127.16.48.62:33593)
0619 17:01:07.157193 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:07.157200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:07.157469 (+ 269us) server_negotiation.cc:407] Connection header received
0619 17:01:07.157670 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:07.157674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:07.157744 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:07.157840 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:07.159829 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:07.160717 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:07.162249 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:07.162996 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:07.163155 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:07.166394 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:07.166404 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:07.166413 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:07.166473 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:07.170704 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:07.172051 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:07.172057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:07.172060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:07.172313 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:07.173004 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:07.173012 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:07.173018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:07.173598 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:07.173963 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:07.174695 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 17:01:07.174906 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 17:01:07.175228 5946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e01f8507e11425a830b28888476fa88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e01f8507e11425a830b28888476fa88" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 33593 } } }
I20260619 17:01:07.175565 5946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:07.178247 5958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:07.183174 5958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:07.193109 5958 catalog_manager.cc:1347] Generated new cluster ID: 167b317a33824070839e19c55cd8e001
I20260619 17:01:07.193313 5958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:07.214886 5958 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:07.216395 5958 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:07.224946 5958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e01f8507e11425a830b28888476fa88: Generated new TSK 0
I20260619 17:01:07.225996 5958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:07.241882 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5868
I20260619 17:01:07.243790 5930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:01:07.338311 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 5868
W20260619 17:01:07.354120 5853 connection.cc:535] client connection to 127.16.48.62:33593 recv error: Network error: failed to read from TLS socket (remote: 127.16.48.62:33593): Connection reset by peer (error 104)
2026-06-19T17:01:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2088 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:07 dist-test-slave-3ffd krb5kdc[5967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:07 dist-test-slave-3ffd krb5kdc[5967](info): set up 2 sockets
Jun 19 17:01:07 dist-test-slave-3ffd krb5kdc[5967](info): commencing operation
krb5kdc: starting...
W20260619 17:01:08.456959 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:08 dist-test-slave-3ffd krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:01:08.736229 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:39107
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:09.084743 5980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:09.085065 5980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:09.085287 5980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:09.092892 5980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:09.093056 5980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:09.093151 5980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:09.093267 5980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:09.093353 5980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:09.103657 5980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:39107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:09.105137 5980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:09.106583 5980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:09.119310 5985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:09.119277 5986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:09.119289 5988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:09.120280 5980 server_base.cc:1034] running on GCE node
I20260619 17:01:09.121112 5980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:09.122609 5980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:09.123888 5980 hybrid_clock.cc:648] HybridClock initialized: now 1781888469123833 us; error 44 us; skew 500 ppm
I20260619 17:01:09.124390 5980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:09 dist-test-slave-3ffd krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888469, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:09.127697 5980 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:01:09.129492 5980 webserver.cc:460] Webserver started at http://127.16.48.62:35619/ using document root <none> and password file <none>
I20260619 17:01:09.130268 5980 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:09.130437 5980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:09.130787 5980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:09.134280 5980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "ef0a4d5ef37043ed816da4326a7cfe96"
format_stamp: "Formatted at 2026-06-19 17:01:09 on dist-test-slave-3ffd"
server_key: "f5b5a25a82063bc7de5415d3ee2022db"
server_key_iv: "d6703926f3d89095f679b9e82e209349"
server_key_version: "encryptionkey@0"
I20260619 17:01:09.135188 5980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "ef0a4d5ef37043ed816da4326a7cfe96"
format_stamp: "Formatted at 2026-06-19 17:01:09 on dist-test-slave-3ffd"
server_key: "f5b5a25a82063bc7de5415d3ee2022db"
server_key_iv: "d6703926f3d89095f679b9e82e209349"
server_key_version: "encryptionkey@0"
I20260619 17:01:09.141052 5980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 17:01:09.145231 5995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:09.146766 5980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 17:01:09.147006 5980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "ef0a4d5ef37043ed816da4326a7cfe96"
format_stamp: "Formatted at 2026-06-19 17:01:09 on dist-test-slave-3ffd"
server_key: "f5b5a25a82063bc7de5415d3ee2022db"
server_key_iv: "d6703926f3d89095f679b9e82e209349"
server_key_version: "encryptionkey@0"
I20260619 17:01:09.147260 5980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:09.170264 5980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:09.175577 5980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:09.175957 5980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:09.197714 5980 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:39107
I20260619 17:01:09.197749 6046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:39107 every 8 connection(s)
I20260619 17:01:09.199292 5980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:09.203675 6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:09.204224 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 5980
I20260619 17:01:09.204404 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:09.204955 16576 external_mini_cluster.cc:1428] Setting key df9f8870a82c11edf47e3ff9c40a08f1
I20260619 17:01:09.219415 6047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96: Bootstrap starting.
I20260619 17:01:09.221074 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:09.207104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33955 (local address 127.16.48.62:39107)
0619 17:01:09.208051 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:09.208064 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:09.208109 (+ 45us) server_negotiation.cc:407] Connection header received
0619 17:01:09.210558 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:09.210593 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:09.211100 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:09.211487 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:09.212231 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:09.213554 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:09.215611 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:09.216556 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:09.216710 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:09.218155 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:09.218163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:09.218180 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:09.219071 (+ 891us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:09.219198 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:09.219252 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:09.219468 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:09.219792 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 17:01:09.219993 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":461,"threads_started":1}
I20260619 17:01:09.223420 6047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:09.224853 6047 log.cc:824] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:09.225616 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:09.227978 6047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96: No bootstrap required, opened a new log
I20260619 17:01:09.237068 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:09.230069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45901 (local address 127.16.48.62:39107)
0619 17:01:09.230245 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:09.230249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:09.230730 (+ 481us) server_negotiation.cc:407] Connection header received
0619 17:01:09.230968 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:09.230974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:09.231054 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:09.231181 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:09.231741 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:09.232653 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:09.234053 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:09.234576 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:09.234770 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:09.236053 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:09.236057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:09.236061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:09.236081 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:09.236179 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:09.236202 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:09.236342 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:09.236546 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 17:01:09.236604 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 17:01:09.236752 6047 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a4d5ef37043ed816da4326a7cfe96" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39107 } }
I20260619 17:01:09.237337 6047 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:01:09.237457 6047 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4d5ef37043ed816da4326a7cfe96, State: Initialized, Role: FOLLOWER
I20260619 17:01:09.238124 6047 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a4d5ef37043ed816da4326a7cfe96" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39107 } }
I20260619 17:01:09.238384 6047 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:01:09.238540 6047 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:01:09.238727 6047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:09.241111 6047 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a4d5ef37043ed816da4326a7cfe96" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39107 } }
I20260619 17:01:09.241675 6047 leader_election.cc:302] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef0a4d5ef37043ed816da4326a7cfe96; no voters:
I20260619 17:01:09.242270 6047 leader_election.cc:288] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:01:09.242429 6058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:01:09.243631 6058 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4d5ef37043ed816da4326a7cfe96, State: Running, Role: LEADER
I20260619 17:01:09.244272 6058 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a4d5ef37043ed816da4326a7cfe96" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39107 } }
I20260619 17:01:09.244828 6047 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:09.247303 6060 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef0a4d5ef37043ed816da4326a7cfe96. Latest consensus state: current_term: 1 leader_uuid: "ef0a4d5ef37043ed816da4326a7cfe96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a4d5ef37043ed816da4326a7cfe96" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39107 } } }
I20260619 17:01:09.247839 6060 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:09.247181 6059 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef0a4d5ef37043ed816da4326a7cfe96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a4d5ef37043ed816da4326a7cfe96" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 39107 } } }
I20260619 17:01:09.248167 6059 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:09.250645 6065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:09.256601 6065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:09.267885 6065 catalog_manager.cc:1347] Generated new cluster ID: b0b30be754df4c50ad37e74c4db4ab60
I20260619 17:01:09.268028 6065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:09.279930 6065 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:09.282038 6065 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:09.290623 6065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef0a4d5ef37043ed816da4326a7cfe96: Generated new TSK 0
I20260619 17:01:09.291787 6065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:09.298388 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5980
W20260619 17:01:09.300637 6042 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33955
W20260619 17:01:09.301518 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 5980: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:01:09.301697 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 5980
W20260619 17:01:09.316159 5965 connection.cc:535] client connection to 127.16.48.62:39107 recv error: Network error: failed to read from TLS socket (remote: 127.16.48.62:39107): Connection reset by peer (error 104)
2026-06-19T17:01:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:09 dist-test-slave-3ffd krb5kdc[6079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:09 dist-test-slave-3ffd krb5kdc[6079](info): set up 2 sockets
Jun 19 17:01:09 dist-test-slave-3ffd krb5kdc[6079](info): commencing operation
krb5kdc: starting...
W20260619 17:01:10.412917 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:10 dist-test-slave-3ffd krb5kdc[6079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:01:10.691713 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42153
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:42153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:11.048559 6092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:11.048870 6092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:11.048964 6092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:11.056825 6092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:11.056972 6092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:11.057022 6092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:11.057080 6092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:11.057128 6092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:11.068011 6092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:42153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:11.069399 6092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:11.070717 6092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:11.083575 6098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:11.083618 6100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:11.083592 6097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:11.084462 6092 server_base.cc:1034] running on GCE node
I20260619 17:01:11.085314 6092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:11.086804 6092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:11.088056 6092 hybrid_clock.cc:648] HybridClock initialized: now 1781888471088011 us; error 34 us; skew 500 ppm
I20260619 17:01:11.088518 6092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:11 dist-test-slave-3ffd krb5kdc[6079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888471, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:11.092018 6092 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:01:11.093780 6092 webserver.cc:460] Webserver started at http://127.16.48.62:42817/ using document root <none> and password file <none>
I20260619 17:01:11.094606 6092 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:11.094832 6092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:11.095257 6092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:11.098564 6092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "33196bd7dce0462ba40a5068e9ddbe2c"
format_stamp: "Formatted at 2026-06-19 17:01:11 on dist-test-slave-3ffd"
server_key: "64e93c63f8129ae97df82809a39838e6"
server_key_iv: "1afea6bb3d6188d0e5c254a672872420"
server_key_version: "encryptionkey@0"
I20260619 17:01:11.099429 6092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "33196bd7dce0462ba40a5068e9ddbe2c"
format_stamp: "Formatted at 2026-06-19 17:01:11 on dist-test-slave-3ffd"
server_key: "64e93c63f8129ae97df82809a39838e6"
server_key_iv: "1afea6bb3d6188d0e5c254a672872420"
server_key_version: "encryptionkey@0"
I20260619 17:01:11.105551 6092 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 17:01:11.109612 6107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:11.111112 6092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 17:01:11.111349 6092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "33196bd7dce0462ba40a5068e9ddbe2c"
format_stamp: "Formatted at 2026-06-19 17:01:11 on dist-test-slave-3ffd"
server_key: "64e93c63f8129ae97df82809a39838e6"
server_key_iv: "1afea6bb3d6188d0e5c254a672872420"
server_key_version: "encryptionkey@0"
I20260619 17:01:11.111677 6092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:11.117993 6092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:11.122614 6092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:11.122962 6092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:11.142808 6092 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:42153
I20260619 17:01:11.142846 6158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:42153 every 8 connection(s)
I20260619 17:01:11.144512 6092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:11.148784 6159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:11.152801 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 6092
I20260619 17:01:11.153026 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:11.153472 16576 external_mini_cluster.cc:1428] Setting key 4ec31649d238b0c357d2022389b212cc
I20260619 17:01:11.162258 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:11.155610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40213 (local address 127.16.48.62:42153)
0619 17:01:11.156534 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:11.156546 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:11.156591 (+ 45us) server_negotiation.cc:407] Connection header received
0619 17:01:11.158300 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:11.158326 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:11.158766 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:11.159151 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:11.160489 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:11.160496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:11.160511 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:11.160530 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:11.160599 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:11.160627 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:11.160817 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:11.160944 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 17:01:11.161210 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":441,"threads_started":1}
I20260619 17:01:11.162899 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c: Bootstrap starting.
I20260619 17:01:11.166584 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:11.166693 6159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:11.167863 6159 log.cc:824] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:11.171701 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c: No bootstrap required, opened a new log
I20260619 17:01:11.174091 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:11.171199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50959 (local address 127.16.48.62:42153)
0619 17:01:11.171425 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:11.171434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:11.171885 (+ 451us) server_negotiation.cc:407] Connection header received
0619 17:01:11.172107 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:11.172164 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:11.172232 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:11.172358 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:11.173358 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:11.173364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:11.173370 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:11.173402 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:11.173445 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:11.173461 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:11.173627 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:11.173716 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 17:01:11.173792 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 17:01:11.180799 6159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33196bd7dce0462ba40a5068e9ddbe2c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42153 } }
I20260619 17:01:11.181286 6159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:01:11.181406 6159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33196bd7dce0462ba40a5068e9ddbe2c, State: Initialized, Role: FOLLOWER
I20260619 17:01:11.182163 6159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33196bd7dce0462ba40a5068e9ddbe2c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42153 } }
I20260619 17:01:11.182438 6159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:01:11.182605 6159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:01:11.182796 6159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:11.185500 6159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33196bd7dce0462ba40a5068e9ddbe2c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42153 } }
I20260619 17:01:11.186185 6159 leader_election.cc:302] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33196bd7dce0462ba40a5068e9ddbe2c; no voters:
I20260619 17:01:11.187011 6159 leader_election.cc:288] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:01:11.187144 6169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:01:11.188197 6169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [term 1 LEADER]: Becoming Leader. State: Replica: 33196bd7dce0462ba40a5068e9ddbe2c, State: Running, Role: LEADER
I20260619 17:01:11.188848 6169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33196bd7dce0462ba40a5068e9ddbe2c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42153 } }
I20260619 17:01:11.190968 6159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:11.192039 6170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33196bd7dce0462ba40a5068e9ddbe2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33196bd7dce0462ba40a5068e9ddbe2c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42153 } } }
I20260619 17:01:11.192536 6170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:11.192931 6171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33196bd7dce0462ba40a5068e9ddbe2c. Latest consensus state: current_term: 1 leader_uuid: "33196bd7dce0462ba40a5068e9ddbe2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33196bd7dce0462ba40a5068e9ddbe2c" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 42153 } } }
I20260619 17:01:11.193970 6171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:11.197430 6176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:11.202961 6176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:11.213975 6176 catalog_manager.cc:1347] Generated new cluster ID: 74277d32eedd433496aed7ac0020ef87
I20260619 17:01:11.214119 6176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:11.230162 6176 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:11.230769 6176 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:11.242336 6176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33196bd7dce0462ba40a5068e9ddbe2c: Generated new TSK 0
I20260619 17:01:11.243304 6176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:11.247675 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6092
W20260619 17:01:11.248842 6154 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40213
W20260619 17:01:11.249583 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6092: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:01:11.249747 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 6092
2026-06-19T17:01:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:11 dist-test-slave-3ffd krb5kdc[6190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:11 dist-test-slave-3ffd krb5kdc[6190](info): set up 2 sockets
Jun 19 17:01:11 dist-test-slave-3ffd krb5kdc[6190](info): commencing operation
krb5kdc: starting...
W20260619 17:01:12.409044 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:12 dist-test-slave-3ffd krb5kdc[6190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:01:12.687906 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46279
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:46279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:13.051946 6203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:13.052295 6203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:13.052503 6203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:13.060034 6203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:13.060256 6203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:13.060362 6203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:13.060477 6203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:13.060542 6203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:13.071148 6203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:46279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:46279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:13.072810 6203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:13.074245 6203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:13.087529 6208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:13.087733 6211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:13.088413 6209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:13.088459 6203 server_base.cc:1034] running on GCE node
I20260619 17:01:13.089346 6203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:13.090929 6203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:13.092268 6203 hybrid_clock.cc:648] HybridClock initialized: now 1781888473092232 us; error 61 us; skew 500 ppm
I20260619 17:01:13.092689 6203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:13 dist-test-slave-3ffd krb5kdc[6190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888473, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:13.101549 6203 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:01:13.103430 6203 webserver.cc:460] Webserver started at http://127.16.48.62:39455/ using document root <none> and password file <none>
I20260619 17:01:13.104305 6203 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:13.104478 6203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:13.104815 6203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:13.108292 6203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "1ef5fb987bf24ac3835df1815dd4dde5"
format_stamp: "Formatted at 2026-06-19 17:01:13 on dist-test-slave-3ffd"
server_key: "3aa02718b063a42f8b71ade5dcc4fb16"
server_key_iv: "29805dfeee432a424e3bf81b0d7657b4"
server_key_version: "encryptionkey@0"
I20260619 17:01:13.109189 6203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "1ef5fb987bf24ac3835df1815dd4dde5"
format_stamp: "Formatted at 2026-06-19 17:01:13 on dist-test-slave-3ffd"
server_key: "3aa02718b063a42f8b71ade5dcc4fb16"
server_key_iv: "29805dfeee432a424e3bf81b0d7657b4"
server_key_version: "encryptionkey@0"
I20260619 17:01:13.114919 6203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 17:01:13.119024 6218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:13.120880 6203 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 17:01:13.121126 6203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "1ef5fb987bf24ac3835df1815dd4dde5"
format_stamp: "Formatted at 2026-06-19 17:01:13 on dist-test-slave-3ffd"
server_key: "3aa02718b063a42f8b71ade5dcc4fb16"
server_key_iv: "29805dfeee432a424e3bf81b0d7657b4"
server_key_version: "encryptionkey@0"
I20260619 17:01:13.121517 6203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:13.135135 6203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:13.140807 6203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:13.141160 6203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:13.161967 6203 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:46279
I20260619 17:01:13.162029 6269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:46279 every 8 connection(s)
I20260619 17:01:13.163604 6203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:13.168426 6270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:13.171203 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 6203
I20260619 17:01:13.171417 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:13.172025 16576 external_mini_cluster.cc:1428] Setting key 108a0d329a498e05a15b87cff6eed13c
I20260619 17:01:13.184027 6270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5: Bootstrap starting.
I20260619 17:01:13.188489 6270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:13.189725 6270 log.cc:824] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5: Log is configured to *not* fsync() on all Append() calls
Jun 19 17:01:13 dist-test-slave-3ffd krb5kdc[6190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:01:13.193965 6270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5: No bootstrap required, opened a new log
I20260619 17:01:13.201563 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:13.174659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41937 (local address 127.16.48.62:46279)
0619 17:01:13.175794 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:13.175814 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:13.176316 (+ 502us) server_negotiation.cc:407] Connection header received
0619 17:01:13.179098 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:13.179137 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:13.179800 (+ 663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:13.180357 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:13.181780 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:13.183866 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:13.185528 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:13.186764 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:13.187009 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:13.191850 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:13.191862 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:13.191885 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:13.191939 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:13.196564 (+ 4625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:13.197499 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:13.197512 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:13.197524 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:13.197825 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:13.198396 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:13.198401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:13.198405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:13.199033 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:13.199366 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:13.199899 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 17:01:13.200408 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":468,"threads_started":1}
I20260619 17:01:13.203711 6270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef5fb987bf24ac3835df1815dd4dde5" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46279 } }
I20260619 17:01:13.204191 6270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:01:13.204339 6270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef5fb987bf24ac3835df1815dd4dde5, State: Initialized, Role: FOLLOWER
I20260619 17:01:13.205010 6270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef5fb987bf24ac3835df1815dd4dde5" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46279 } }
I20260619 17:01:13.205317 6270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:01:13.205467 6270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:01:13.205673 6270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:13.205926 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:13.208065 6270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef5fb987bf24ac3835df1815dd4dde5" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46279 } }
I20260619 17:01:13.208760 6270 leader_election.cc:302] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ef5fb987bf24ac3835df1815dd4dde5; no voters:
I20260619 17:01:13.209568 6270 leader_election.cc:288] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:01:13.209657 6279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:01:13.212410 6270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:13.217743 6279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [term 1 LEADER]: Becoming Leader. State: Replica: 1ef5fb987bf24ac3835df1815dd4dde5, State: Running, Role: LEADER
I20260619 17:01:13.218715 6279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef5fb987bf24ac3835df1815dd4dde5" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46279 } }
I20260619 17:01:13.222177 6280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ef5fb987bf24ac3835df1815dd4dde5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef5fb987bf24ac3835df1815dd4dde5" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46279 } } }
I20260619 17:01:13.222609 6280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:13.223253 6281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ef5fb987bf24ac3835df1815dd4dde5. Latest consensus state: current_term: 1 leader_uuid: "1ef5fb987bf24ac3835df1815dd4dde5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef5fb987bf24ac3835df1815dd4dde5" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 46279 } } }
I20260619 17:01:13.223582 6281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:13.223619 6286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:13.236608 6286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:13.237021 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:13.211046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51781 (local address 127.16.48.62:46279)
0619 17:01:13.216466 (+ 5420us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:13.216476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:13.217145 (+ 669us) server_negotiation.cc:407] Connection header received
0619 17:01:13.217374 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:13.217381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:13.217464 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:13.217594 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:13.219977 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:13.220998 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:13.222971 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:13.224011 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:13.225041 (+ 1030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:13.228359 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:13.228369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:13.228377 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:13.228439 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:13.232610 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:13.233694 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:13.233702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:13.233708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:13.233879 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:13.234391 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:13.234395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:13.234399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:13.234864 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:13.236208 (+ 1344us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:13.236395 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 17:01:13.236670 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5188}
I20260619 17:01:13.248270 6286 catalog_manager.cc:1347] Generated new cluster ID: cdf5d937d4f34ecfa9921db8354addd7
I20260619 17:01:13.248413 6286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:13.272012 6286 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:13.274081 6286 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:13.283421 6286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ef5fb987bf24ac3835df1815dd4dde5: Generated new TSK 0
I20260619 17:01:13.284677 6286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:13.293195 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6203
I20260619 17:01:13.294400 6265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:01:13.388449 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 6203
2026-06-19T17:01:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2139 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:13 dist-test-slave-3ffd krb5kdc[6302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:13 dist-test-slave-3ffd krb5kdc[6302](info): set up 2 sockets
Jun 19 17:01:13 dist-test-slave-3ffd krb5kdc[6302](info): commencing operation
krb5kdc: starting...
W20260619 17:01:14.505023 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:14 dist-test-slave-3ffd krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:01:14.787508 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:32773
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:32773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:15.147781 6315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:15.148285 6315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:15.148483 6315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:15.163540 6315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:15.163757 6315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:15.163867 6315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:15.163972 6315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:15.164053 6315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:15.179636 6315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:32773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:32773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:15.181080 6315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:15.182428 6315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:15.195447 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:15.195571 6320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:15.195596 6321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:15.196743 6315 server_base.cc:1034] running on GCE node
I20260619 17:01:15.197652 6315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:15.199261 6315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:15.200513 6315 hybrid_clock.cc:648] HybridClock initialized: now 1781888475200461 us; error 74 us; skew 500 ppm
I20260619 17:01:15.200942 6315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:15 dist-test-slave-3ffd krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888475, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:15.204830 6315 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:01:15.207013 6315 webserver.cc:460] Webserver started at http://127.16.48.62:45839/ using document root <none> and password file <none>
I20260619 17:01:15.207793 6315 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:15.207937 6315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:15.208302 6315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:15.211493 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "676eff9009594e38938b3143fab30e6d"
format_stamp: "Formatted at 2026-06-19 17:01:15 on dist-test-slave-3ffd"
server_key: "ed5a92818c692e9e7037b6f0892b57cf"
server_key_iv: "f0c91eccff9cab27d597dbe46efec032"
server_key_version: "encryptionkey@0"
I20260619 17:01:15.212435 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "676eff9009594e38938b3143fab30e6d"
format_stamp: "Formatted at 2026-06-19 17:01:15 on dist-test-slave-3ffd"
server_key: "ed5a92818c692e9e7037b6f0892b57cf"
server_key_iv: "f0c91eccff9cab27d597dbe46efec032"
server_key_version: "encryptionkey@0"
I20260619 17:01:15.218062 6315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 17:01:15.221858 6330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:15.223340 6315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 17:01:15.223551 6315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "676eff9009594e38938b3143fab30e6d"
format_stamp: "Formatted at 2026-06-19 17:01:15 on dist-test-slave-3ffd"
server_key: "ed5a92818c692e9e7037b6f0892b57cf"
server_key_iv: "f0c91eccff9cab27d597dbe46efec032"
server_key_version: "encryptionkey@0"
I20260619 17:01:15.223778 6315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:15.247416 6315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:15.252514 6315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:15.252842 6315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:15.272869 6315 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:32773
I20260619 17:01:15.272903 6381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:32773 every 8 connection(s)
I20260619 17:01:15.274400 6315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:15.275645 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 6315
I20260619 17:01:15.275847 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:15.276356 16576 external_mini_cluster.cc:1428] Setting key c770b8aba64304b45a1d9cdaa3017de5
I20260619 17:01:15.279803 6382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:15.291054 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:15.278328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46485 (local address 127.16.48.62:32773)
0619 17:01:15.279176 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:15.279189 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:15.279233 (+ 44us) server_negotiation.cc:407] Connection header received
0619 17:01:15.281111 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:15.281137 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:15.281579 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:15.281938 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:15.282656 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:15.284217 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:15.286162 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:15.286926 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:15.287080 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:15.288478 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:15.288486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:15.288502 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:15.289108 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:15.289212 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:15.289242 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:15.289336 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:15.289626 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 17:01:15.289957 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":374,"threads_started":1}
I20260619 17:01:15.293458 6382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d: Bootstrap starting.
I20260619 17:01:15.295074 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:15.297377 6382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:15.298921 6382 log.cc:824] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:15.302742 6382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d: No bootstrap required, opened a new log
I20260619 17:01:15.307788 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:15.300598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50029 (local address 127.16.48.62:32773)
0619 17:01:15.300777 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:15.300783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:15.301193 (+ 410us) server_negotiation.cc:407] Connection header received
0619 17:01:15.301358 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:15.301364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:15.301454 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:15.301596 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:15.302122 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:15.303180 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:15.304846 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:15.305596 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:15.305759 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:15.306904 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:15.306910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:15.306914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:15.306940 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:15.306991 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:15.307011 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:15.307128 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:15.307308 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 17:01:15.307416 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 17:01:15.312605 6382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676eff9009594e38938b3143fab30e6d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 32773 } }
I20260619 17:01:15.313228 6382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:01:15.313375 6382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676eff9009594e38938b3143fab30e6d, State: Initialized, Role: FOLLOWER
I20260619 17:01:15.314018 6382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676eff9009594e38938b3143fab30e6d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 32773 } }
I20260619 17:01:15.314302 6382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:01:15.314457 6382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:01:15.314636 6382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:15.317409 6382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676eff9009594e38938b3143fab30e6d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 32773 } }
I20260619 17:01:15.318022 6382 leader_election.cc:302] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 676eff9009594e38938b3143fab30e6d; no voters:
I20260619 17:01:15.318657 6382 leader_election.cc:288] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:01:15.318837 6392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:01:15.319594 6392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [term 1 LEADER]: Becoming Leader. State: Replica: 676eff9009594e38938b3143fab30e6d, State: Running, Role: LEADER
I20260619 17:01:15.320238 6392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676eff9009594e38938b3143fab30e6d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 32773 } }
I20260619 17:01:15.320994 6382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:15.323320 6394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 676eff9009594e38938b3143fab30e6d. Latest consensus state: current_term: 1 leader_uuid: "676eff9009594e38938b3143fab30e6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676eff9009594e38938b3143fab30e6d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 32773 } } }
I20260619 17:01:15.323457 6393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "676eff9009594e38938b3143fab30e6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676eff9009594e38938b3143fab30e6d" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 32773 } } }
I20260619 17:01:15.323769 6394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:15.323896 6393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:15.335924 6397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:15.341454 6397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:15.350981 6397 catalog_manager.cc:1347] Generated new cluster ID: 42a73917178f40fc87ed90f51a131d22
I20260619 17:01:15.351102 6397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:15.368330 6397 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:15.369940 6397 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:15.381927 6397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 676eff9009594e38938b3143fab30e6d: Generated new TSK 0
I20260619 17:01:15.382895 6397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:15.390288 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6315
W20260619 17:01:15.391604 6377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46485
W20260619 17:01:15.392377 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6315: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:01:15.392532 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 6315
2026-06-19T17:01:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:15 dist-test-slave-3ffd krb5kdc[6413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:15 dist-test-slave-3ffd krb5kdc[6413](info): set up 2 sockets
Jun 19 17:01:15 dist-test-slave-3ffd krb5kdc[6413](info): commencing operation
krb5kdc: starting...
W20260619 17:01:16.509060 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:16 dist-test-slave-3ffd krb5kdc[6413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:01:16.791812 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:40755
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:40755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:17.143158 6426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:17.143492 6426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:17.143663 6426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:17.151957 6426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:17.152105 6426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:17.152215 6426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:17.152318 6426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:17.152410 6426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:17.163007 6426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:40755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:17.164501 6426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:17.166000 6426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:17.179303 6431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:17.179265 6432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:17.179559 6434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:17.180513 6426 server_base.cc:1034] running on GCE node
I20260619 17:01:17.181512 6426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:17.183146 6426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:17.184405 6426 hybrid_clock.cc:648] HybridClock initialized: now 1781888477184369 us; error 62 us; skew 500 ppm
I20260619 17:01:17.184804 6426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:17 dist-test-slave-3ffd krb5kdc[6413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888477, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:17.188397 6426 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:01:17.190043 6426 webserver.cc:460] Webserver started at http://127.16.48.62:46261/ using document root <none> and password file <none>
I20260619 17:01:17.190841 6426 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:17.191009 6426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:17.191356 6426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:17.194732 6426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "a727e99181ad4190aa3bdca24086aff4"
format_stamp: "Formatted at 2026-06-19 17:01:17 on dist-test-slave-3ffd"
server_key: "bb8a1dd6a3714d5a30a2ef98c7900ab9"
server_key_iv: "b4594fb48549209ed20058129ad5fee1"
server_key_version: "encryptionkey@0"
I20260619 17:01:17.195637 6426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "a727e99181ad4190aa3bdca24086aff4"
format_stamp: "Formatted at 2026-06-19 17:01:17 on dist-test-slave-3ffd"
server_key: "bb8a1dd6a3714d5a30a2ef98c7900ab9"
server_key_iv: "b4594fb48549209ed20058129ad5fee1"
server_key_version: "encryptionkey@0"
I20260619 17:01:17.201501 6426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 17:01:17.205613 6441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:17.207193 6426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 17:01:17.207412 6426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "a727e99181ad4190aa3bdca24086aff4"
format_stamp: "Formatted at 2026-06-19 17:01:17 on dist-test-slave-3ffd"
server_key: "bb8a1dd6a3714d5a30a2ef98c7900ab9"
server_key_iv: "b4594fb48549209ed20058129ad5fee1"
server_key_version: "encryptionkey@0"
I20260619 17:01:17.207655 6426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:17.214751 6426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:17.219743 6426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:17.220019 6426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:17.239768 6426 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:40755
I20260619 17:01:17.239799 6492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:40755 every 8 connection(s)
I20260619 17:01:17.241421 6426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:17.245589 6493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:17.249755 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 6426
I20260619 17:01:17.249960 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:17.250392 16576 external_mini_cluster.cc:1428] Setting key 91a037fc895b67701a88c5b2edba2093
I20260619 17:01:17.258828 6493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4: Bootstrap starting.
I20260619 17:01:17.260789 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:17.252454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43625 (local address 127.16.48.62:40755)
0619 17:01:17.253366 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:17.253381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:17.253445 (+ 64us) server_negotiation.cc:407] Connection header received
0619 17:01:17.256692 (+ 3247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:17.256728 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:17.257263 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:17.257747 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:17.258767 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:17.258775 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:17.258791 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:17.258817 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:17.258891 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:17.258921 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:17.259155 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:17.259283 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 17:01:17.259487 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":404,"threads_started":1}
I20260619 17:01:17.262828 6493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:17.264240 6493 log.cc:824] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:17.265170 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:17.267305 6493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4: No bootstrap required, opened a new log
I20260619 17:01:17.272274 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:17.269613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41033 (local address 127.16.48.62:40755)
0619 17:01:17.269768 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:17.269772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:17.270294 (+ 522us) server_negotiation.cc:407] Connection header received
0619 17:01:17.270473 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:17.270477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:17.270528 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:17.270617 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:17.271558 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:17.271562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:17.271565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 17:01:17.271579 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:17.271616 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 17:01:17.271631 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:17.271730 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:17.271865 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 17:01:17.271912 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 17:01:17.276505 6493 raft_consensus.cc:348] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a727e99181ad4190aa3bdca24086aff4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40755 } }
I20260619 17:01:17.276932 6493 raft_consensus.cc:374] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 17:01:17.277055 6493 raft_consensus.cc:729] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a727e99181ad4190aa3bdca24086aff4, State: Initialized, Role: FOLLOWER
I20260619 17:01:17.277731 6493 consensus_queue.cc:260] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a727e99181ad4190aa3bdca24086aff4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40755 } }
I20260619 17:01:17.277985 6493 raft_consensus.cc:388] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 17:01:17.278148 6493 raft_consensus.cc:482] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 17:01:17.278331 6493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:17.280721 6493 raft_consensus.cc:504] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a727e99181ad4190aa3bdca24086aff4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40755 } }
I20260619 17:01:17.281294 6493 leader_election.cc:302] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a727e99181ad4190aa3bdca24086aff4; no voters:
I20260619 17:01:17.281951 6493 leader_election.cc:288] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 17:01:17.282161 6503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 17:01:17.283434 6503 raft_consensus.cc:686] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [term 1 LEADER]: Becoming Leader. State: Replica: a727e99181ad4190aa3bdca24086aff4, State: Running, Role: LEADER
I20260619 17:01:17.284044 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a727e99181ad4190aa3bdca24086aff4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40755 } }
I20260619 17:01:17.284494 6493 sys_catalog.cc:564] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:17.286978 6504 sys_catalog.cc:455] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a727e99181ad4190aa3bdca24086aff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a727e99181ad4190aa3bdca24086aff4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40755 } } }
I20260619 17:01:17.287401 6504 sys_catalog.cc:458] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:17.287093 6505 sys_catalog.cc:455] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a727e99181ad4190aa3bdca24086aff4. Latest consensus state: current_term: 1 leader_uuid: "a727e99181ad4190aa3bdca24086aff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a727e99181ad4190aa3bdca24086aff4" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 40755 } } }
I20260619 17:01:17.287515 6505 sys_catalog.cc:458] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:17.290536 6509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:17.296469 6509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:17.305835 6509 catalog_manager.cc:1347] Generated new cluster ID: f83ca1d9ccd249b8a39d00f7f666cc22
I20260619 17:01:17.305968 6509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:17.327981 6509 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:17.328573 6509 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:17.337308 6509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a727e99181ad4190aa3bdca24086aff4: Generated new TSK 0
I20260619 17:01:17.338507 6509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:17.344892 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6426
W20260619 17:01:17.346131 6488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43625
W20260619 17:01:17.346894 16576 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6426: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 17:01:17.347071 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 6426
2026-06-19T17:01:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1951 ms)
[----------] 18 tests from AuthTokenIssuingTest (36487 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:17 dist-test-slave-3ffd krb5kdc[6524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:17 dist-test-slave-3ffd krb5kdc[6524](info): set up 2 sockets
Jun 19 17:01:17 dist-test-slave-3ffd krb5kdc[6524](info): commencing operation
krb5kdc: starting...
W20260619 17:01:18.460987 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:18 dist-test-slave-3ffd krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:01:18.743934 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:37999
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:37999,127.16.48.61:38589,127.16.48.60:42841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:19.100383 6537 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 17:01:19.100780 6537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:19.100899 6537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:19.101037 6537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:19.108677 6537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:19.108795 6537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:19.108902 6537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:19.108992 6537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:19.109094 6537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:19.119563 6537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:37999,127.16.48.61:38589,127.16.48.60:42841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:19.121132 6537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:19.122578 6537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:19.135535 6543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:19.135564 6542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:19.135586 6545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:19.136265 6537 server_base.cc:1034] running on GCE node
I20260619 17:01:19.137311 6537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:19.138830 6537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:19.140175 6537 hybrid_clock.cc:648] HybridClock initialized: now 1781888479140021 us; error 59 us; skew 500 ppm
I20260619 17:01:19.140573 6537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:19 dist-test-slave-3ffd krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:19.150691 6537 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:01:19.152529 6537 webserver.cc:460] Webserver started at http://127.16.48.62:34121/ using document root <none> and password file <none>
I20260619 17:01:19.153332 6537 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:19.153481 6537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:19.153825 6537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:19.157121 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "bb188aac9de847369f200804a0ed9229"
format_stamp: "Formatted at 2026-06-19 17:01:19 on dist-test-slave-3ffd"
server_key: "ae71993b690168f958dda0e6ed1512aa"
server_key_iv: "3ad2a9aa23c3e0e566ca41416e1a2b24"
server_key_version: "encryptionkey@0"
I20260619 17:01:19.157991 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "bb188aac9de847369f200804a0ed9229"
format_stamp: "Formatted at 2026-06-19 17:01:19 on dist-test-slave-3ffd"
server_key: "ae71993b690168f958dda0e6ed1512aa"
server_key_iv: "3ad2a9aa23c3e0e566ca41416e1a2b24"
server_key_version: "encryptionkey@0"
I20260619 17:01:19.163709 6537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 17:01:19.167770 6552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:19.169387 6537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 17:01:19.169620 6537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "bb188aac9de847369f200804a0ed9229"
format_stamp: "Formatted at 2026-06-19 17:01:19 on dist-test-slave-3ffd"
server_key: "ae71993b690168f958dda0e6ed1512aa"
server_key_iv: "3ad2a9aa23c3e0e566ca41416e1a2b24"
server_key_version: "encryptionkey@0"
I20260619 17:01:19.169924 6537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:19.187306 6537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:19.191978 6537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:19.192344 6537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:19.213052 6537 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:37999
I20260619 17:01:19.213090 6603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:37999 every 8 connection(s)
I20260619 17:01:19.214527 6537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:19.218461 6604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:19.222595 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 6537
I20260619 17:01:19.222883 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:19.223692 16576 external_mini_cluster.cc:1428] Setting key 845bb311432b42d372f78accc73f3880
I20260619 17:01:19.227389 6604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 17:01:19 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.48.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.61@KRBTEST.COM" created.
I20260619 17:01:19.256070 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.232724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.16.48.62:37999)
0619 17:01:19.233452 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:19.233458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:19.233504 (+ 46us) server_negotiation.cc:407] Connection header received
0619 17:01:19.236061 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:19.236067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:19.236456 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:19.236672 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:19.238504 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.239789 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.242230 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.242882 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.243041 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:19.247873 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:19.247883 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:19.247897 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:19.247933 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:19.251141 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:19.251960 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:19.251973 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:19.251976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:19.252199 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:19.252627 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:19.252631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:19.252635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:19.253199 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:19.253429 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:19.253998 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 17:01:19.254480 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":543,"thread_start_us":393,"threads_started":1}
I20260619 17:01:19.256098 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.231236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37999 (local address 127.0.0.1:43014)
0619 17:01:19.232296 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.232359 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:19.235544 (+ 3185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:19.236759 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:19.236769 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:19.237675 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:19.238237 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:19.238268 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.240001 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.240009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:19.242036 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:19.242057 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.243167 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.243177 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:19.243425 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:19.244235 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:19.244273 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:19.247556 (+ 3283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:19.251333 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:19.251348 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:19.251362 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:19.251798 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:19.252314 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:19.252321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:19.252326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:19.252497 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:19.253359 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:19.253367 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:19.253587 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 17:01:19.253857 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 17:01:19.254614 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":242,"thread_start_us":399,"threads_started":1}
I20260619 17:01:19.259502 6604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } has no permanent_uuid. Determining permanent_uuid...
I20260619 17:01:19.260769 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.260329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33038)
0619 17:01:19.260510 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.260639 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 17:01:19.261071 6553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
I20260619 17:01:19.262977 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.262525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33044)
0619 17:01:19.262746 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.262826 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260619 17:01:19.263190 6604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:38589: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab.
Entry for principal kudu/127.16.48.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab.
I20260619 17:01:19.287937 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.61:38589
--webserver_interface=127.16.48.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab
--principal=kudu/127.16.48.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:37999,127.16.48.61:38589,127.16.48.60:42841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 17:01:19.312570 6604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } attempt: 1
I20260619 17:01:19.313876 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.313300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33056)
0619 17:01:19.313552 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.313702 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 17:01:19.315119 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.314667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33070)
0619 17:01:19.314832 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.314928 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 17:01:19.315358 6604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:38589: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
I20260619 17:01:19.383661 6604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } attempt: 2
I20260619 17:01:19.385144 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.384496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33082)
0619 17:01:19.384816 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.384932 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 17:01:19.386427 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.385927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33096)
0619 17:01:19.386121 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.386236 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 17:01:19.386672 6604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:38589: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
I20260619 17:01:19.477936 6604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } attempt: 3
I20260619 17:01:19.479138 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.478676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33098)
0619 17:01:19.478883 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.478985 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 17:01:19.480305 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.479837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33110)
0619 17:01:19.479994 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.480101 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 17:01:19.480484 6604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:38589: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
I20260619 17:01:19.623755 6604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } attempt: 4
I20260619 17:01:19.625177 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.624496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33118)
0619 17:01:19.624869 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.624981 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260619 17:01:19.626366 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.625930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33120)
0619 17:01:19.626099 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.626206 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 17:01:19.626603 6604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:38589: Network error: Client connection negotiation failed: client connection to 127.16.48.61:38589: connect: Connection refused (error 111)
W20260619 17:01:19.643905 6611 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 17:01:19.644418 6611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:19.644542 6611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:19.644680 6611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:19.652527 6611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:19.652673 6611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:19.652783 6611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:19.652868 6611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:19.652967 6611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:19.663439 6611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:37999,127.16.48.61:38589,127.16.48.60:42841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab
--principal=kudu/127.16.48.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.61:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.48.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:19.664906 6611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:19.666313 6611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:19.679340 6616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:19.679289 6617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:19.679314 6619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:19.679807 6611 server_base.cc:1034] running on GCE node
I20260619 17:01:19.680912 6611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:19.682437 6611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:19.683684 6611 hybrid_clock.cc:648] HybridClock initialized: now 1781888479683617 us; error 36 us; skew 500 ppm
I20260619 17:01:19.684062 6611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:19 dist-test-slave-3ffd krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:19.687626 6611 init.cc:377] Logged in from keytab as kudu/127.16.48.61@KRBTEST.COM (short username kudu)
I20260619 17:01:19.689311 6611 webserver.cc:460] Webserver started at http://127.16.48.61:39555/ using document root <none> and password file <none>
I20260619 17:01:19.690145 6611 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:19.690290 6611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:19.690654 6611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:19.694213 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/data/instance:
uuid: "324104bafa5b4cfda9cfbf62c097a534"
format_stamp: "Formatted at 2026-06-19 17:01:19 on dist-test-slave-3ffd"
server_key: "2b2323e4792c9361a4b719aa1b98f548"
server_key_iv: "e5a3227f7c71268300fefdebb8ddde8f"
server_key_version: "encryptionkey@0"
I20260619 17:01:19.695271 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/wal/instance:
uuid: "324104bafa5b4cfda9cfbf62c097a534"
format_stamp: "Formatted at 2026-06-19 17:01:19 on dist-test-slave-3ffd"
server_key: "2b2323e4792c9361a4b719aa1b98f548"
server_key_iv: "e5a3227f7c71268300fefdebb8ddde8f"
server_key_version: "encryptionkey@0"
I20260619 17:01:19.701443 6611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 17:01:19.705576 6626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:19.707207 6611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 17:01:19.707443 6611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/wal
uuid: "324104bafa5b4cfda9cfbf62c097a534"
format_stamp: "Formatted at 2026-06-19 17:01:19 on dist-test-slave-3ffd"
server_key: "2b2323e4792c9361a4b719aa1b98f548"
server_key_iv: "e5a3227f7c71268300fefdebb8ddde8f"
server_key_version: "encryptionkey@0"
I20260619 17:01:19.707718 6611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:19.728653 6611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:19.733767 6611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:19.734104 6611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:19.754043 6611 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.61:38589
I20260619 17:01:19.754058 6677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.61:38589 every 8 connection(s)
I20260619 17:01:19.755686 6611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
I20260619 17:01:19.760309 6678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:19.765758 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 6611
I20260619 17:01:19.765990 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-1/wal/instance
I20260619 17:01:19.766412 16576 external_mini_cluster.cc:1428] Setting key 010909ce5306b94b8e9d338031b2df62
I20260619 17:01:19.767603 6678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 17:01:19 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:01:19.790117 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.770622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.16.48.62:37999)
0619 17:01:19.771168 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:19.771173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:19.772266 (+ 1093us) server_negotiation.cc:407] Connection header received
0619 17:01:19.774880 (+ 2614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:19.774887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:19.774985 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:19.775115 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:19.776911 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.777623 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.779252 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.779682 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.779828 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:19.783635 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:19.783640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:19.783643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:19.783691 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:19.786628 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:19.787527 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:19.787532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:19.787534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:19.787673 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:19.788192 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:19.788196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:19.788198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:19.788498 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:19.788678 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:19.789444 (+ 766us) server_negotiation.cc:299] Negotiation successful
0619 17:01:19.789691 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":305,"threads_started":1}
WARNING: no policy specified for kudu/127.16.48.60@KRBTEST.COM; defaulting to no policy
I20260619 17:01:19.791219 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.771231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37999 (local address 127.0.0.1:43018)
0619 17:01:19.772040 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.772080 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:19.774512 (+ 2432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:19.775404 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:19.775414 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:19.775974 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:19.776671 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:19.776690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.777809 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.777813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:19.779085 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:19.779100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.779945 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.779953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:19.780281 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:19.781046 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:19.781072 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:19.783340 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:19.786830 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:19.786842 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:19.786862 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:19.787352 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:19.787798 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:19.787803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:19.787806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:19.788016 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:19.788647 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:19.788656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:19.789058 (+ 402us) client_negotiation.cc:769] Sending connection context
0619 17:01:19.789350 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 17:01:19.789937 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":335,"threads_started":1}
I20260619 17:01:19.793294 6678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.60@KRBTEST.COM" created.
Jun 19 17:01:19 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 17:01:19.810304 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.794110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33134)
0619 17:01:19.794613 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.794631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:19.794777 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:19.795599 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:19.795603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:19.796060 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:19.796263 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:19.796272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.797638 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.797647 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:19.798510 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:19.798521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.799409 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.799415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:19.799669 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:19.800384 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:19.800402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:19.802863 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:19.807698 (+ 4835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:19.807703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:19.807706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:19.808093 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:19.808596 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:19.808599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:19.808602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:19.808743 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:19.809537 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:19.809540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:19.809681 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 17:01:19.809785 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 17:01:19.810014 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":50}
I20260619 17:01:19.810846 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.794509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.16.48.61:38589)
0619 17:01:19.795149 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:19.795154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:19.795180 (+ 26us) server_negotiation.cc:407] Connection header received
0619 17:01:19.795228 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:19.795234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:19.795430 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:19.795602 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:19.796447 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.797487 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.798672 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.799251 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.799396 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:19.803146 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:19.803161 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:19.803177 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:19.803210 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:19.807483 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:19.808295 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:19.808309 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:19.808315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:19.808478 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:19.808877 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:19.808885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:19.808890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:19.809414 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:19.809631 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:19.810017 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 17:01:19.810318 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":486,"thread_start_us":328,"threads_started":1}
I20260619 17:01:19.813601 6678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
I20260619 17:01:19.814921 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.814439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54080)
0619 17:01:19.814634 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.814757 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 17:01:19.815366 6629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
I20260619 17:01:19.817173 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.816803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54092)
0619 17:01:19.816953 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.817044 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 17:01:19.817528 6678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:42841: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab.
Entry for principal kudu/127.16.48.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab.
I20260619 17:01:19.830974 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.60:42841
--webserver_interface=127.16.48.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab
--principal=kudu/127.16.48.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:37999,127.16.48.61:38589,127.16.48.60:42841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 17:01:19.866952 6678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } attempt: 1
I20260619 17:01:19.868271 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.867721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54096)
0619 17:01:19.867948 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.868041 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 17:01:19.869315 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.868968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54112)
0619 17:01:19.869110 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.869207 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 17:01:19.869561 6678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:42841: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
I20260619 17:01:19.925944 6604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } attempt: 5
Jun 19 17:01:19 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 17:01:19.937819 6678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } attempt: 2
I20260619 17:01:19.939085 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.938525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54116)
0619 17:01:19.938773 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.938871 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 17:01:19.940340 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.939904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54130)
0619 17:01:19.940089 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.940215 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 17:01:19.940610 6678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:42841: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
I20260619 17:01:19.941216 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.927122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.16.48.61:38589)
0619 17:01:19.927358 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:19.927373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:19.927389 (+ 16us) server_negotiation.cc:407] Connection header received
0619 17:01:19.927443 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:19.927447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:19.927507 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:19.927632 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:19.928593 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.929307 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.930635 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.931158 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.931308 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:19.934982 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:19.934988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:19.934992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:19.935031 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:19.937941 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:19.938832 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:19.938837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:19.938839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:19.938982 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:19.939547 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:19.939553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:19.939556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:19.939920 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:19.940108 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:19.940681 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 17:01:19.940974 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 17:01:19.941216 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.926775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33140)
0619 17:01:19.927055 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.927085 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:19.927240 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:19.927670 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:19.927676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:19.928208 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:19.928409 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:19.928420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.929468 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.929475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:19.930443 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:19.930457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:19.931327 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:19.931334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:19.931546 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:19.932272 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:19.932291 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:19.934691 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:19.938156 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:19.938163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:19.938170 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:19.938646 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:19.939149 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:19.939159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:19.939165 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:19.939395 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:19.940166 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:19.940171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:19.940354 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 17:01:19.940522 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 17:01:19.940868 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 17:01:19.942308 6604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
I20260619 17:01:19.943523 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.943038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54132)
0619 17:01:19.943242 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.943340 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 17:01:19.944684 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.944282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54136)
0619 17:01:19.944441 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.944557 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 17:01:19.944890 6604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:42841: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
I20260619 17:01:19.996192 6604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } attempt: 1
I20260619 17:01:19.997632 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.996936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54144)
0619 17:01:19.997211 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.997357 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 17:01:19.999027 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:19.998529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54146)
0619 17:01:19.998745 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:19.998855 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 17:01:19.999300 6604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:42841: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
I20260619 17:01:20.031867 6678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } attempt: 3
I20260619 17:01:20.033205 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.032682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54156)
0619 17:01:20.032934 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.033029 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 17:01:20.034371 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.033926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54172)
0619 17:01:20.034155 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.034244 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260619 17:01:20.034596 6678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:42841: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
I20260619 17:01:20.067585 6604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } attempt: 2
I20260619 17:01:20.069026 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.068391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54182)
0619 17:01:20.068691 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.068809 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 17:01:20.070264 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.069760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54192)
0619 17:01:20.069966 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.070076 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 17:01:20.070534 6604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:42841: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
I20260619 17:01:20.176807 6604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } attempt: 3
I20260619 17:01:20.177824 6678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } attempt: 4
I20260619 17:01:20.178148 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.177577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54196)
0619 17:01:20.177819 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.177948 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 17:01:20.178983 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.178475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54202)
0619 17:01:20.178692 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.178798 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 17:01:20.179301 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.178808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54206)
0619 17:01:20.179065 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.179155 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260619 17:01:20.179517 6604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:42841: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
I20260619 17:01:20.180085 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.179683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54222)
0619 17:01:20.179853 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.179947 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 17:01:20.180449 6678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:42841: Network error: Client connection negotiation failed: client connection to 127.16.48.60:42841: connect: Connection refused (error 111)
W20260619 17:01:20.193645 6686 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 17:01:20.194038 6686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:20.194141 6686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:20.194309 6686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:20.202253 6686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:20.202375 6686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:20.202440 6686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:20.202487 6686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:20.202605 6686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:20.213380 6686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:37999,127.16.48.61:38589,127.16.48.60:42841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab
--principal=kudu/127.16.48.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.60:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.48.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:20.214748 6686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:20.216199 6686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:20.230207 6694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:20.230261 6691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:20.230238 6692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:20.231410 6686 server_base.cc:1034] running on GCE node
I20260619 17:01:20.232420 6686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:20.234015 6686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:20.235283 6686 hybrid_clock.cc:648] HybridClock initialized: now 1781888480235217 us; error 49 us; skew 500 ppm
I20260619 17:01:20.235642 6686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:20 dist-test-slave-3ffd krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888480, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:20.239107 6686 init.cc:377] Logged in from keytab as kudu/127.16.48.60@KRBTEST.COM (short username kudu)
I20260619 17:01:20.240809 6686 webserver.cc:460] Webserver started at http://127.16.48.60:44609/ using document root <none> and password file <none>
I20260619 17:01:20.241575 6686 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:20.241712 6686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:20.242062 6686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:20.245285 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/data/instance:
uuid: "e7e49041a56947b587a3d6ca21b0bb9c"
format_stamp: "Formatted at 2026-06-19 17:01:20 on dist-test-slave-3ffd"
server_key: "5462929c59a5c31b4b1691650b30f050"
server_key_iv: "43bef28e029d9a592ae3bef57c7d2897"
server_key_version: "encryptionkey@0"
I20260619 17:01:20.246178 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/wal/instance:
uuid: "e7e49041a56947b587a3d6ca21b0bb9c"
format_stamp: "Formatted at 2026-06-19 17:01:20 on dist-test-slave-3ffd"
server_key: "5462929c59a5c31b4b1691650b30f050"
server_key_iv: "43bef28e029d9a592ae3bef57c7d2897"
server_key_version: "encryptionkey@0"
I20260619 17:01:20.251965 6686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 17:01:20.255964 6701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:20.257539 6686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 17:01:20.257774 6686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/wal
uuid: "e7e49041a56947b587a3d6ca21b0bb9c"
format_stamp: "Formatted at 2026-06-19 17:01:20 on dist-test-slave-3ffd"
server_key: "5462929c59a5c31b4b1691650b30f050"
server_key_iv: "43bef28e029d9a592ae3bef57c7d2897"
server_key_version: "encryptionkey@0"
I20260619 17:01:20.258044 6686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:20.295703 6686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:20.300164 6686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:20.300504 6686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:20.320204 6686 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.60:42841
I20260619 17:01:20.320170 6753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.60:42841 every 8 connection(s)
I20260619 17:01:20.321722 6686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
I20260619 17:01:20.325968 6754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:20.328553 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 6686
I20260619 17:01:20.328763 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781888107067647-16576-0/minicluster-data/master-2/wal/instance
I20260619 17:01:20.329190 16576 external_mini_cluster.cc:1428] Setting key 7e48b8b6738fe931613cbb4f211ada7a
I20260619 17:01:20.334429 6754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 17:01:20 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
Jun 19 17:01:20 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888480, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:01:20.354346 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.331353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.16.48.60:42841)
0619 17:01:20.332300 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.332312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.332355 (+ 43us) server_negotiation.cc:407] Connection header received
0619 17:01:20.334093 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.334117 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.334575 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.335054 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.336642 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.338618 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.340693 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.341532 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.341732 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.346613 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.346628 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.346651 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.346696 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.349654 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.350478 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.350486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.350494 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.350686 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.351141 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.351146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.351149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.351797 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.352182 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.352722 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.353196 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":679,"thread_start_us":416,"threads_started":1}
I20260619 17:01:20.354739 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.337174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.16.48.62:37999)
0619 17:01:20.337766 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.337774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.338307 (+ 533us) server_negotiation.cc:407] Connection header received
0619 17:01:20.338481 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.338486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.338572 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.338667 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.340346 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.341169 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.342918 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.343670 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.343825 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.347796 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.347802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.347805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.347852 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.351509 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.352428 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.352434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.352438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.352595 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.353070 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.353076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.353079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.353385 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.353558 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.354072 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.354314 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":265,"threads_started":1}
I20260619 17:01:20.354779 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.337197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:37999 (local address 127.0.0.1:43022)
0619 17:01:20.338138 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.338189 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:20.338358 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:20.338772 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:20.338782 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:20.339715 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:20.340139 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.340155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.341339 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.341344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.342747 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.342758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.343900 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.343904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.344034 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.344757 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:20.344793 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:20.347545 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:20.351677 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.351691 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.351700 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.352283 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.352719 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.352725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.352729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.352891 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.353527 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:20.353535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:20.353722 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 17:01:20.353918 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 17:01:20.354271 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":45,"thread_start_us":518,"threads_started":1}
I20260619 17:01:20.356325 6754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } has no permanent_uuid. Determining permanent_uuid...
I20260619 17:01:20.356748 6604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } attempt: 4
Jun 19 17:01:20 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
Jun 19 17:01:20 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888480, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
Jun 19 17:01:20 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:01:20.373627 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.357609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.16.48.60:42841)
0619 17:01:20.357767 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.357773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.357788 (+ 15us) server_negotiation.cc:407] Connection header received
0619 17:01:20.357959 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.357964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.358031 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.358140 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.359144 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.359964 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.361552 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.362378 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.362630 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.366869 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.366876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.366881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.366915 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.370237 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.371068 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.371074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.371078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.371261 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.371809 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.371814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.371820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.372263 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.372410 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.372938 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.373177 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":55}
I20260619 17:01:20.373644 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.357320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54246)
0619 17:01:20.357688 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.357706 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:20.357810 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:20.358172 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:20.358176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:20.358762 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:20.358986 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.359000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.360111 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.360148 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.361391 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.361410 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.362650 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.362656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.362875 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.363638 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:20.363663 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:20.366605 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:20.370395 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.370401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.370406 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.370892 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.371424 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.371430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.371435 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.371649 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.372479 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:20.372487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:20.372673 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 17:01:20.372846 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 17:01:20.373226 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260619 17:01:20.374819 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.357491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.16.48.61:38589)
0619 17:01:20.357667 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.357674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.357696 (+ 22us) server_negotiation.cc:407] Connection header received
0619 17:01:20.357765 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.357771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.357851 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.357958 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.359484 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.360648 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.362371 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.363090 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.363254 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.367197 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.367206 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.367214 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.367265 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.371068 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.372259 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.372265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.372269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.372436 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.372945 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.372954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.372957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.373354 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.373490 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.374174 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.374418 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 17:01:20.375813 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.357072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:38589 (local address 127.0.0.1:33144)
0619 17:01:20.357287 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.357309 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:20.357418 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:20.358007 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:20.358014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:20.359095 (+ 1081us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:20.359300 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.359310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.360827 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.360833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.362181 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.362198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.363603 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.363608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.363808 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.364438 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:20.364463 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:20.366926 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:20.371250 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.371255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.371258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.371792 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.372580 (+ 788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.372587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.372592 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.372785 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.373600 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:20.373612 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:20.373787 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 17:01:20.375150 (+ 1363us) client_negotiation.cc:240] Negotiation successful
0619 17:01:20.375475 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":28}
I20260619 17:01:20.377269 6754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
I20260619 17:01:20.377611 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.358779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.16.48.62:37999)
0619 17:01:20.358958 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.358963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.358981 (+ 18us) server_negotiation.cc:407] Connection header received
0619 17:01:20.359035 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.359042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.359147 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.359235 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.360981 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.361987 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.363795 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.364639 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.364782 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.370158 (+ 5376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.370166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.370173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.370223 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.373780 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.374752 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.374757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.374760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.374916 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.375385 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.375392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.375395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.375754 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.376086 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.377020 (+ 934us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.377294 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 17:01:20.379918 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229: Bootstrap starting.
I20260619 17:01:20.384792 6604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:20.386490 6604 log.cc:824] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229: Log is configured to *not* fsync() on all Append() calls
Jun 19 17:01:20 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888480, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
Jun 19 17:01:20 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 17:01:20.390520 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229: No bootstrap required, opened a new log
I20260619 17:01:20.396004 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.380573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.16.48.60:42841)
0619 17:01:20.380797 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.380806 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.381192 (+ 386us) server_negotiation.cc:407] Connection header received
0619 17:01:20.381237 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.381241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.381308 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.381388 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.382262 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.383026 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.384711 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.385324 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.385470 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.388882 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.388889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.388894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.388932 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.392626 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.393529 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.393536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.393542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.393699 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.394114 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.394120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.394123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.394574 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.394711 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.395290 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.395626 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":50}
I20260619 17:01:20.396615 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.380580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54254)
0619 17:01:20.380922 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.380942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:20.381087 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:20.381547 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:20.381551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:20.381981 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:20.382120 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.382130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.383180 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.383185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.384534 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.384555 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.385487 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.385493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.385699 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.386269 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:20.386287 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:20.388611 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:20.392810 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.392816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.392821 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.393375 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.393830 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.393836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.393840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.393991 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.394798 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:20.394803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:20.394958 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 17:01:20.395157 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 17:01:20.396308 (+ 1151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260619 17:01:20.396631 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.379409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.16.48.61:38589)
0619 17:01:20.379593 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.379599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.379619 (+ 20us) server_negotiation.cc:407] Connection header received
0619 17:01:20.379683 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.379687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.379760 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.379869 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.381728 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.382910 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.384273 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.384780 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.384913 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.390380 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.390386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.390389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.390428 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.393679 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.394433 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.394438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.394440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.394588 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.395005 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.395017 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.395020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.395382 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.395704 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.395993 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.396262 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 17:01:20.402086 6604 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } }
I20260619 17:01:20.402508 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:20.402571 6604 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb188aac9de847369f200804a0ed9229, State: Initialized, Role: FOLLOWER
I20260619 17:01:20.402614 6754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c: Bootstrap starting.
I20260619 17:01:20.403081 6604 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } }
I20260619 17:01:20.404106 6761 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.404660 6761 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:20.405550 6604 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:20.406378 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bb188aac9de847369f200804a0ed9229"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43038
I20260619 17:01:20.406621 6754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:20.406800 6579 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 17:01:20.407049 6579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:20.408046 6754 log.cc:824] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:20.410569 6579 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } }
I20260619 17:01:20.412031 6754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c: No bootstrap required, opened a new log
I20260619 17:01:20.414044 6579 leader_election.cc:288] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [CANDIDATE]: Term 1 election: Requested vote from peers 324104bafa5b4cfda9cfbf62c097a534 (127.16.48.61:38589), e7e49041a56947b587a3d6ca21b0bb9c (127.16.48.60:42841)
I20260619 17:01:20.425770 6754 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } }
I20260619 17:01:20.426592 6754 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e49041a56947b587a3d6ca21b0bb9c, State: Initialized, Role: FOLLOWER
I20260619 17:01:20.427431 6754 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } }
W20260619 17:01:20.434643 6774 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 17:01:20.434882 6774 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 17:01:20.435173 6754 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:20.435322 6729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb188aac9de847369f200804a0ed9229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e7e49041a56947b587a3d6ca21b0bb9c"
I20260619 17:01:20.435500 6777 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.436010 6777 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:20.436087 6729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:20.441191 6729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb188aac9de847369f200804a0ed9229 in term 1.
I20260619 17:01:20.442202 6555 leader_election.cc:302] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb188aac9de847369f200804a0ed9229, e7e49041a56947b587a3d6ca21b0bb9c; no voters:
I20260619 17:01:20.442844 6761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [term 1 FOLLOWER]: Leader election won for term 1
W20260619 17:01:20.445606 6558 tablet.cc:2369] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:01:20.459515 6761 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [term 1 LEADER]: Becoming Leader. State: Replica: bb188aac9de847369f200804a0ed9229, State: Running, Role: LEADER
I20260619 17:01:20.425704 6537 master_runner.cc:186] Error getting master registration for 127.16.48.61:38589: OK, instance_id { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" instance_seqno: 1781888479748328 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 17:01:20.460593 6761 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } }
I20260619 17:01:20.465270 6537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 17:01:20.479746 6678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } attempt: 5
I20260619 17:01:20.478832 6781 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb188aac9de847369f200804a0ed9229. Latest consensus state: current_term: 1 leader_uuid: "bb188aac9de847369f200804a0ed9229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.481505 6781 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:20.484875 6789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:20.487480 6789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:20.488289 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.443764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.16.48.62:37999)
0619 17:01:20.444003 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.444012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.454027 (+ 10015us) server_negotiation.cc:407] Connection header received
0619 17:01:20.455214 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.455224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.455347 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.455444 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.463236 (+ 7792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.464452 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.473127 (+ 8675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.473865 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.473985 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.477844 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.477853 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.477860 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.477949 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.482808 (+ 4859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.484148 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.484156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.484163 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.484397 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.485036 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.485043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.485049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.485691 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.486097 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.487405 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.487691 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 17:01:20.500850 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.450337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.16.48.60:42841)
0619 17:01:20.450543 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.450549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.456833 (+ 6284us) server_negotiation.cc:407] Connection header received
0619 17:01:20.458424 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.458434 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.458594 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.458690 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.463260 (+ 4570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.464452 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.469441 (+ 4989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.470230 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.470348 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.491359 (+ 21011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.491368 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.491373 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.491423 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.495461 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.497325 (+ 1864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.497333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.497336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.497502 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.498327 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.498333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.498337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.498725 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.499004 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.500135 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.500448 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 17:01:20 dist-test-slave-3ffd krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
I20260619 17:01:20.511759 6729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [term 1 FOLLOWER]: Refusing update from remote peer bb188aac9de847369f200804a0ed9229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:01:20.513195 6781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:01:20.529076 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.457538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.16.48.61:38589)
0619 17:01:20.457740 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.457745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.463132 (+ 5387us) server_negotiation.cc:407] Connection header received
0619 17:01:20.466337 (+ 3205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.466345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.466450 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.466536 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.471450 (+ 4914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.472644 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.478615 (+ 5971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.479422 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.479539 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.518944 (+ 39405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.518952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.518957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.519011 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.522958 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.524243 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.524252 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.524257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.524456 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.525795 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.525800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.525804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.526230 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.526508 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.528322 (+ 1814us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.528612 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 17:01:20.532558 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.480410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:42841 (local address 127.0.0.1:54264)
0619 17:01:20.480841 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:20.480864 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:20.480984 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:20.501556 (+ 20572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:20.501563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:20.502115 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:20.502360 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.502377 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.503996 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.504003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.505530 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:20.505548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.506735 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.506742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:20.507055 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.507839 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:20.507866 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:20.510978 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:20.525025 (+ 14047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.525031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.525035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.525640 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.527536 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:20.527554 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:20.527560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:20.527741 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:20.530943 (+ 3202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:20.530948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:20.531133 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 17:01:20.531275 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 17:01:20.532097 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260619 17:01:20.533970 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.481251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54264 (local address 127.16.48.60:42841)
0619 17:01:20.501125 (+ 19874us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.501132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.501150 (+ 18us) server_negotiation.cc:407] Connection header received
0619 17:01:20.501204 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.501211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.501311 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.502532 (+ 1221us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.502725 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.503786 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.505802 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.506537 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.511199 (+ 4662us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.511500 (+ 301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:20.511506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:20.511510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:20.511542 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:20.518634 (+ 7092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.526655 (+ 8021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.526661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.526664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.526868 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:20.527918 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:20.527925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:20.527929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:20.530782 (+ 2853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:20.533018 (+ 2236us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.533213 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.533519 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19753,"thread_start_us":33353,"threads_started":1}
I20260619 17:01:20.541244 6678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534: Bootstrap starting.
I20260619 17:01:20.548271 6678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:20.549894 6678 log.cc:824] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534: Log is configured to *not* fsync() on all Append() calls
W20260619 17:01:20.552973 6708 tablet.cc:2369] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:01:20.560849 6777 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb188aac9de847369f200804a0ed9229. Latest consensus state: current_term: 1 leader_uuid: "bb188aac9de847369f200804a0ed9229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.561211 6777 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:20.562702 6678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534: No bootstrap required, opened a new log
I20260619 17:01:20.567123 6797 mvcc.cc:205] Tried to move back new op lower bound from 7298615216159760384 to 7298615215982051328. Current Snapshot: MvccSnapshot[applied={T|T < 7298615216159760384}]
I20260619 17:01:20.564973 6801 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 17:01:20.573349 6777 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb188aac9de847369f200804a0ed9229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.573769 6777 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:20.577788 6789 catalog_manager.cc:1347] Generated new cluster ID: d7ba54dc0f914853be7c89fab30c85b5
I20260619 17:01:20.577924 6789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:20.577984 6678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } }
I20260619 17:01:20.578449 6678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324104bafa5b4cfda9cfbf62c097a534, State: Initialized, Role: FOLLOWER
I20260619 17:01:20.579016 6678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } }
I20260619 17:01:20.580539 6781 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb188aac9de847369f200804a0ed9229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.580828 6781 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:20.586141 6804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.586602 6804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:20.588613 6678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:20.589107 6652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:20.592367 6653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb188aac9de847369f200804a0ed9229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "324104bafa5b4cfda9cfbf62c097a534"
I20260619 17:01:20.593194 6652 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [term 1 FOLLOWER]: Refusing update from remote peer bb188aac9de847369f200804a0ed9229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:01:20.595113 6761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:01:20.599004 6653 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb188aac9de847369f200804a0ed9229 in term 1.
I20260619 17:01:20.613267 6789 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:20.615127 6789 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:20.586222 6686 master_runner.cc:186] Error getting master registration for 127.16.48.61:38589: OK, instance_id { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" instance_seqno: 1781888479748328 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 17:01:20.617156 6686 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 17:01:20.628578 6789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229: Generated new TSK 0
I20260619 17:01:20.629717 6789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:20.632871 6804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb188aac9de847369f200804a0ed9229. Latest consensus state: current_term: 1 leader_uuid: "bb188aac9de847369f200804a0ed9229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.633283 6804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:20.632473 6811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb188aac9de847369f200804a0ed9229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.633424 6811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:20.634498 6761 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb188aac9de847369f200804a0ed9229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb188aac9de847369f200804a0ed9229" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 37999 } } peers { permanent_uuid: "324104bafa5b4cfda9cfbf62c097a534" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 38589 } } peers { permanent_uuid: "e7e49041a56947b587a3d6ca21b0bb9c" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 42841 } } }
I20260619 17:01:20.634835 6761 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb188aac9de847369f200804a0ed9229 [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:20.683701 6822 catalog_manager.cc:1259] Loaded cluster ID: d7ba54dc0f914853be7c89fab30c85b5
I20260619 17:01:20.683841 6822 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534: loading cluster ID for follower catalog manager: success
I20260619 17:01:20.688110 6822 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534: acquiring CA information for follower catalog manager: success
I20260619 17:01:20.691359 6822 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 324104bafa5b4cfda9cfbf62c097a534: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 17:01:20.722182 6758 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.718629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.16.48.62:37999)
0619 17:01:20.718863 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.718870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.720106 (+ 1236us) server_negotiation.cc:407] Connection header received
0619 17:01:20.720472 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.720479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.720577 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.720663 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.721884 (+ 1221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43070: BlockingRecv error: recv got EOF from 127.0.0.1:43070 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 17:01:20.724843 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.719658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.16.48.60:42841)
0619 17:01:20.719854 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.719861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.722954 (+ 3093us) server_negotiation.cc:407] Connection header received
0619 17:01:20.723255 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.723261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.723350 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.723430 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.724558 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54272: BlockingRecv error: recv got EOF from 127.0.0.1:54272 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 17:01:20.731371 6683 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.729229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.16.48.61:38589)
0619 17:01:20.729516 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.729524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.729573 (+ 49us) server_negotiation.cc:407] Connection header received
0619 17:01:20.729797 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.729803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.729894 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.730011 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.731086 (+ 1075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33166: BlockingRecv error: recv got EOF from 127.0.0.1:33166 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 17:01:20.743651 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.737124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.16.48.61:38589)
0619 17:01:20.737305 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.737315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.737889 (+ 574us) server_negotiation.cc:407] Connection header received
0619 17:01:20.738144 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.738153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.738364 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.738538 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:20.738977 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.739733 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.741177 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.741637 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.741781 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.742321 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:20.742924 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:20.743112 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.743166 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.743266 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 17:01:20.753077 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.750694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.16.48.60:42841)
0619 17:01:20.750969 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.750979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.751498 (+ 519us) server_negotiation.cc:407] Connection header received
0619 17:01:20.751773 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.751778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.751887 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.752012 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.752819 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54284: BlockingRecv error: recv got EOF from 127.0.0.1:54284 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 17:01:20.765077 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.757784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.16.48.61:38589)
0619 17:01:20.758070 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.758080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.758600 (+ 520us) server_negotiation.cc:407] Connection header received
0619 17:01:20.758820 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.758826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.758907 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.759011 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:20.759471 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.760375 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.762593 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.763324 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.763453 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.764138 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:20.764325 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:20.764404 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.764648 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.764740 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260619 17:01:20.787237 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.772411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54290 (local address 127.16.48.60:42841)
0619 17:01:20.772606 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.772612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.785374 (+ 12762us) server_negotiation.cc:407] Connection header received
0619 17:01:20.785653 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.785660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.785764 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.785840 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.787006 (+ 1166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54290: BlockingRecv error: recv got EOF from 127.0.0.1:54290 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 17:01:20.803023 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.793219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.16.48.61:38589)
0619 17:01:20.793437 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.793444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.796547 (+ 3103us) server_negotiation.cc:407] Connection header received
0619 17:01:20.796678 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.796685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.796781 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.796891 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:20.797404 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.798624 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.800649 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.801356 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.801536 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.802083 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:20.802255 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:20.802353 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.802539 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.802637 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 17:01:20.814122 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.811601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.16.48.60:42841)
0619 17:01:20.811806 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.811813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.812398 (+ 585us) server_negotiation.cc:407] Connection header received
0619 17:01:20.812694 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.812700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.812832 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.812951 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.813865 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54300: BlockingRecv error: recv got EOF from 127.0.0.1:54300 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 17:01:20.834383 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.826573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.16.48.61:38589)
0619 17:01:20.826829 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.826836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.827369 (+ 533us) server_negotiation.cc:407] Connection header received
0619 17:01:20.827573 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.827584 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.827686 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.827795 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:20.828320 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.829662 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.831871 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.832755 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.832893 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.833496 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:20.833658 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:20.833755 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.833958 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.834054 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260619 17:01:20.850394 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.846137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.16.48.60:42841)
0619 17:01:20.846412 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.846419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.848778 (+ 2359us) server_negotiation.cc:407] Connection header received
0619 17:01:20.848973 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.848979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.849086 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.849204 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.850149 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54306: BlockingRecv error: recv got EOF from 127.0.0.1:54306 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
I20260619 17:01:20.869621 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.862022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.16.48.61:38589)
0619 17:01:20.862225 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.862231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.862813 (+ 582us) server_negotiation.cc:407] Connection header received
0619 17:01:20.863157 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.863165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.863265 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.863380 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:20.863919 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.865170 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.867342 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.868058 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.868229 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.868776 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:20.868918 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:20.869010 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.869184 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.869287 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 17:01:20.885181 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.882661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.16.48.60:42841)
0619 17:01:20.882854 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.882860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.883454 (+ 594us) server_negotiation.cc:407] Connection header received
0619 17:01:20.883670 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.883677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.883778 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.883907 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.884866 (+ 959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54316: BlockingRecv error: recv got EOF from 127.0.0.1:54316 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 17:01:20.913461 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.905250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.16.48.61:38589)
0619 17:01:20.905457 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.905464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.906056 (+ 592us) server_negotiation.cc:407] Connection header received
0619 17:01:20.906263 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.906269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.906371 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.906510 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:20.907051 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.908365 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.910494 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.911392 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.911604 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.912495 (+ 891us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:20.912692 (+ 197us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:20.912840 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.912999 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.913143 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 17:01:20.923089 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.920716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.16.48.60:42841)
0619 17:01:20.920951 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.920958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.921509 (+ 551us) server_negotiation.cc:407] Connection header received
0619 17:01:20.921700 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.921707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.921813 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.921933 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.922843 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54330: BlockingRecv error: recv got EOF from 127.0.0.1:54330 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 17:01:20.969875 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:20.962275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.16.48.61:38589)
0619 17:01:20.962512 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.962521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.963103 (+ 582us) server_negotiation.cc:407] Connection header received
0619 17:01:20.963441 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.963448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.963538 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.963642 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:20.964192 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.965347 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.967473 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:20.968261 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:20.968415 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:20.968948 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:20.969113 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:20.969219 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:20.969417 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 17:01:20.969528 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 17:01:20.984344 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:20.981696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.16.48.60:42841)
0619 17:01:20.981937 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:20.981946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:20.982587 (+ 641us) server_negotiation.cc:407] Connection header received
0619 17:01:20.982807 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:20.982813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:20.982929 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:20.983052 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:20.984042 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54336: BlockingRecv error: recv got EOF from 127.0.0.1:54336 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 17:01:21.065933 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:21.057885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.16.48.61:38589)
0619 17:01:21.058119 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:21.058127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:21.058787 (+ 660us) server_negotiation.cc:407] Connection header received
0619 17:01:21.059032 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:21.059038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:21.059135 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:21.059235 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:21.059933 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:21.061245 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:21.063523 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:21.064300 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:21.064454 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:21.065046 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:21.065223 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:21.065315 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:21.065538 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 17:01:21.065632 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 17:01:21.079049 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:21.076521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.16.48.60:42841)
0619 17:01:21.076793 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:21.076803 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:21.077417 (+ 614us) server_negotiation.cc:407] Connection header received
0619 17:01:21.077745 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:21.077752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:21.077860 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:21.077991 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:21.078783 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54344: BlockingRecv error: recv got EOF from 127.0.0.1:54344 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 17:01:21.216462 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:21.210692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.16.48.61:38589)
0619 17:01:21.210914 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:21.210922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:21.211181 (+ 259us) server_negotiation.cc:407] Connection header received
0619 17:01:21.211430 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:21.211434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:21.211503 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:21.211607 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:21.212187 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:21.213199 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:21.214591 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:21.215046 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:21.215198 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:21.215627 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:21.215747 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:21.215851 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:21.216013 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 17:01:21.216096 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 17:01:21.224421 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:21.222309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.16.48.60:42841)
0619 17:01:21.222596 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:21.222602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:21.222919 (+ 317us) server_negotiation.cc:407] Connection header received
0619 17:01:21.223091 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:21.223096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:21.223173 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:21.223261 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:21.224193 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54354: BlockingRecv error: recv got EOF from 127.0.0.1:54354 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
I20260619 17:01:21.490578 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:21.484917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.16.48.61:38589)
0619 17:01:21.485188 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:21.485196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:21.485461 (+ 265us) server_negotiation.cc:407] Connection header received
0619 17:01:21.485697 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:21.485703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:21.485793 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:21.485903 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:21.486337 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:21.487082 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:21.488616 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:21.489033 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:21.489184 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:21.489653 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:21.489777 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:21.489865 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:21.490148 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 17:01:21.490288 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 17:01:21.498906 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:21.496768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.16.48.60:42841)
0619 17:01:21.497067 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:21.497075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:21.497336 (+ 261us) server_negotiation.cc:407] Connection header received
0619 17:01:21.497572 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:21.497580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:21.497678 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:21.497791 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:21.498715 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54366: BlockingRecv error: recv got EOF from 127.0.0.1:54366 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 17:01:21.572546 6801 catalog_manager.cc:1259] Loaded cluster ID: d7ba54dc0f914853be7c89fab30c85b5
I20260619 17:01:21.572669 6801 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c: loading cluster ID for follower catalog manager: success
I20260619 17:01:21.576061 6801 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c: acquiring CA information for follower catalog manager: success
I20260619 17:01:21.578338 6801 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e7e49041a56947b587a3d6ca21b0bb9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 17:01:22.020291 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:22.014805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.16.48.61:38589)
0619 17:01:22.015330 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:22.015334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:22.015355 (+ 21us) server_negotiation.cc:407] Connection header received
0619 17:01:22.015701 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:22.015705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:22.015777 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:22.015873 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:22.016346 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.017089 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.018471 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.018885 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.018981 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:22.019470 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:22.019586 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:22.019670 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:22.019817 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 17:01:22.019887 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":290,"threads_started":1}
I20260619 17:01:22.032305 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:22.026088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.16.48.60:42841)
0619 17:01:22.026679 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:22.026684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:22.026705 (+ 21us) server_negotiation.cc:407] Connection header received
0619 17:01:22.026930 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:22.026935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:22.027097 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:22.027210 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:22.027619 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.028468 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.029855 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.030304 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.030435 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:22.030982 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:22.031574 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:22.031748 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:22.031832 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 17:01:22.031907 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":332,"threads_started":1}
I20260619 17:01:22.047971 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:22.039429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.16.48.62:37999)
0619 17:01:22.040934 (+ 1505us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:22.040941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:22.040965 (+ 24us) server_negotiation.cc:407] Connection header received
0619 17:01:22.041114 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:22.041121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:22.041373 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:22.041562 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:22.042334 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.043351 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.045383 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.045980 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.046131 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:22.046724 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:22.047264 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:22.047382 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:22.047511 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 17:01:22.047607 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":1263,"threads_started":1}
I20260619 17:01:22.048007 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:22.040400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.16.48.61:38589)
0619 17:01:22.041017 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:22.041026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:22.041794 (+ 768us) server_negotiation.cc:407] Connection header received
0619 17:01:22.041932 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:22.041937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:22.042033 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:22.042136 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:22.042889 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.043935 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.045733 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.046371 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.046543 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:22.047123 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:22.047272 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:22.047360 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:22.047646 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 17:01:22.047732 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437}
I20260619 17:01:22.048727 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:22.039510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.16.48.60:42841)
0619 17:01:22.039748 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:22.039757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:22.041360 (+ 1603us) server_negotiation.cc:407] Connection header received
0619 17:01:22.041724 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:22.041732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:22.041840 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:22.041930 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:22.042715 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.043874 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.046042 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:22.046625 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:22.046761 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:22.047273 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:22.047403 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:22.047491 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:22.048419 (+ 928us) server_negotiation.cc:299] Negotiation successful
0619 17:01:22.048510 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 17:01:22.054383 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6537
I20260619 17:01:22.055510 6599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:01:22.168138 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 6537
I20260619 17:01:22.190179 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6611
I20260619 17:01:22.191325 6673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:01:22.303773 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 6611
I20260619 17:01:22.325035 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6686
I20260619 17:01:22.326293 6749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:01:22.434118 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 6686
2026-06-19T17:01:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5095 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 17:01:22 dist-test-slave-3ffd krb5kdc[6957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 17:01:22 dist-test-slave-3ffd krb5kdc[6957](info): set up 2 sockets
Jun 19 17:01:22 dist-test-slave-3ffd krb5kdc[6957](info): commencing operation
krb5kdc: starting...
W20260619 17:01:23.573086 16576 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.008s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/test-user.keytab.
Jun 19 17:01:23 dist-test-slave-3ffd krb5kdc[6957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T17:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T17:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
Entry for principal kudu/127.16.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab.
I20260619 17:01:23.863723 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:44601
--webserver_interface=127.16.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:44601,127.16.48.61:46211,127.16.48.60:35681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 17:01:24.226082 6970 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 17:01:24.226446 6970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:24.226615 6970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:24.226747 6970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:24.234434 6970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:24.234579 6970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:24.234640 6970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:24.234752 6970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:24.234864 6970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:24.246855 6970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:44601,127.16.48.61:46211,127.16.48.60:35681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.62.keytab
--principal=kudu/127.16.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.62:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:24.248378 6970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:24.249794 6970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:24.267632 6975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:24.267688 6978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:24.267715 6976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:24.268783 6970 server_base.cc:1034] running on GCE node
I20260619 17:01:24.269723 6970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:24.271347 6970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:24.272640 6970 hybrid_clock.cc:648] HybridClock initialized: now 1781888484272573 us; error 46 us; skew 500 ppm
I20260619 17:01:24.273027 6970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:24 dist-test-slave-3ffd krb5kdc[6957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:24.281955 6970 init.cc:377] Logged in from keytab as kudu/127.16.48.62@KRBTEST.COM (short username kudu)
I20260619 17:01:24.283833 6970 webserver.cc:460] Webserver started at http://127.16.48.62:42559/ using document root <none> and password file <none>
I20260619 17:01:24.284744 6970 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:24.284916 6970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:24.285305 6970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:24.288810 6970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/data/instance:
uuid: "3638c182c9bb416da6efbff28026102f"
format_stamp: "Formatted at 2026-06-19 17:01:24 on dist-test-slave-3ffd"
server_key: "7f4eaa1bfd7cd12e047aa10ff8d1ea88"
server_key_iv: "3a8d128290f1fd5ab38fa7724bdced96"
server_key_version: "encryptionkey@0"
I20260619 17:01:24.289775 6970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/wal/instance:
uuid: "3638c182c9bb416da6efbff28026102f"
format_stamp: "Formatted at 2026-06-19 17:01:24 on dist-test-slave-3ffd"
server_key: "7f4eaa1bfd7cd12e047aa10ff8d1ea88"
server_key_iv: "3a8d128290f1fd5ab38fa7724bdced96"
server_key_version: "encryptionkey@0"
I20260619 17:01:24.295871 6970 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 17:01:24.300006 6985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:24.301892 6970 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 17:01:24.302126 6970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/wal
uuid: "3638c182c9bb416da6efbff28026102f"
format_stamp: "Formatted at 2026-06-19 17:01:24 on dist-test-slave-3ffd"
server_key: "7f4eaa1bfd7cd12e047aa10ff8d1ea88"
server_key_iv: "3a8d128290f1fd5ab38fa7724bdced96"
server_key_version: "encryptionkey@0"
I20260619 17:01:24.302392 6970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:24.319454 6970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:24.324590 6970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:24.324908 6970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:24.345558 6970 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.62:44601
I20260619 17:01:24.345605 7036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.62:44601 every 8 connection(s)
I20260619 17:01:24.347097 6970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/data/info.pb
I20260619 17:01:24.351418 7037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:24.352857 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 6970
I20260619 17:01:24.353061 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-0/wal/instance
I20260619 17:01:24.353601 16576 external_mini_cluster.cc:1428] Setting key 55648031d756fb042e508b25d2fbc0a2
I20260619 17:01:24.362684 7037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 17:01:24 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.48.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.61@KRBTEST.COM" created.
I20260619 17:01:24.389094 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.367352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.16.48.62:44601)
0619 17:01:24.368431 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:24.368456 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:24.369030 (+ 574us) server_negotiation.cc:407] Connection header received
0619 17:01:24.371344 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:24.371351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:24.371659 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:24.371903 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:24.373387 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.374638 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.376270 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.376873 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.377103 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:24.380783 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:24.380791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:24.380805 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:24.380837 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:24.384073 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:24.385105 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:24.385113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:24.385115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:24.385296 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:24.385871 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:24.385875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:24.385878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:24.386464 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:24.386673 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:24.387463 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 17:01:24.387788 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":520,"threads_started":1}
I20260619 17:01:24.389218 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.367352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44601 (local address 127.0.0.1:33296)
0619 17:01:24.368819 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.368862 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:24.370845 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:24.371914 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:24.371922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:24.372755 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:24.373126 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:24.373147 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.374820 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.374825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:24.376077 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:24.376096 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.377140 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.377146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:24.377316 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:24.378091 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:24.378116 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:24.380506 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:24.384299 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:24.384318 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:24.384330 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:24.384844 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:24.385448 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:24.385456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:24.385462 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:24.385721 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:24.386664 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:24.386676 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:24.386968 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 17:01:24.387299 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 17:01:24.388036 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"mutex_wait_us":135,"thread_start_us":955,"threads_started":1}
I20260619 17:01:24.392802 7037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } has no permanent_uuid. Determining permanent_uuid...
I20260619 17:01:24.394168 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.393594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36084)
0619 17:01:24.393816 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.393971 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 17:01:24.394533 6987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
I20260619 17:01:24.396565 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.396031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36098)
0619 17:01:24.396279 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.396401 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260619 17:01:24.396773 7037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:46211: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab.
Entry for principal kudu/127.16.48.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab.
I20260619 17:01:24.420012 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.61:46211
--webserver_interface=127.16.48.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab
--principal=kudu/127.16.48.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:44601,127.16.48.61:46211,127.16.48.60:35681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 17:01:24.446156 7037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } attempt: 1
I20260619 17:01:24.447434 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.446939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36112)
0619 17:01:24.447179 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.447278 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 17:01:24.448629 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.448251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36126)
0619 17:01:24.448408 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.448483 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 17:01:24.448813 7037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:46211: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
I20260619 17:01:24.517076 7037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } attempt: 2
I20260619 17:01:24.518396 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.517782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36132)
0619 17:01:24.518090 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.518212 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 17:01:24.519636 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.519165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36138)
0619 17:01:24.519328 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.519440 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 17:01:24.519862 7037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:46211: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
I20260619 17:01:24.611181 7037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } attempt: 3
I20260619 17:01:24.612629 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.612063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36148)
0619 17:01:24.612361 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.612466 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 17:01:24.614012 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.613623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36164)
0619 17:01:24.613780 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.613856 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 17:01:24.614316 7037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:46211: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
I20260619 17:01:24.757640 7037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } attempt: 4
I20260619 17:01:24.759037 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.758451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36180)
0619 17:01:24.758770 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.758898 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 17:01:24.760216 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.759756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36194)
0619 17:01:24.759927 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.760018 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 17:01:24.760417 7037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.61:46211: Network error: Client connection negotiation failed: client connection to 127.16.48.61:46211: connect: Connection refused (error 111)
W20260619 17:01:24.781926 7044 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 17:01:24.782374 7044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:24.782517 7044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:24.782667 7044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:24.790876 7044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:24.791050 7044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:24.791165 7044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:24.791265 7044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:24.791356 7044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:24.802095 7044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:44601,127.16.48.61:46211,127.16.48.60:35681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.61.keytab
--principal=kudu/127.16.48.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.61:46211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.48.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:24.803556 7044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:24.805094 7044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:24.818821 7049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:24.818815 7052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:24.818852 7050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:24.819885 7044 server_base.cc:1034] running on GCE node
I20260619 17:01:24.820819 7044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:24.822372 7044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:24.823661 7044 hybrid_clock.cc:648] HybridClock initialized: now 1781888484823601 us; error 39 us; skew 500 ppm
I20260619 17:01:24.823994 7044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:24 dist-test-slave-3ffd krb5kdc[6957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:24.827734 7044 init.cc:377] Logged in from keytab as kudu/127.16.48.61@KRBTEST.COM (short username kudu)
I20260619 17:01:24.829670 7044 webserver.cc:460] Webserver started at http://127.16.48.61:42663/ using document root <none> and password file <none>
I20260619 17:01:24.830471 7044 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:24.830606 7044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:24.830964 7044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:24.834405 7044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/data/instance:
uuid: "03aad3aeace2409687ee768ac0365e4c"
format_stamp: "Formatted at 2026-06-19 17:01:24 on dist-test-slave-3ffd"
server_key: "fb7963eea096920b287d025475c5fc22"
server_key_iv: "ea78fab15b99f20d01053fc928ee2a89"
server_key_version: "encryptionkey@0"
I20260619 17:01:24.835415 7044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/wal/instance:
uuid: "03aad3aeace2409687ee768ac0365e4c"
format_stamp: "Formatted at 2026-06-19 17:01:24 on dist-test-slave-3ffd"
server_key: "fb7963eea096920b287d025475c5fc22"
server_key_iv: "ea78fab15b99f20d01053fc928ee2a89"
server_key_version: "encryptionkey@0"
I20260619 17:01:24.841589 7044 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260619 17:01:24.845695 7059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:24.847271 7044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 17:01:24.847506 7044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/wal
uuid: "03aad3aeace2409687ee768ac0365e4c"
format_stamp: "Formatted at 2026-06-19 17:01:24 on dist-test-slave-3ffd"
server_key: "fb7963eea096920b287d025475c5fc22"
server_key_iv: "ea78fab15b99f20d01053fc928ee2a89"
server_key_version: "encryptionkey@0"
I20260619 17:01:24.847759 7044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:24.872156 7044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:24.876911 7044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:24.877247 7044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:24.897282 7044 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.61:46211
I20260619 17:01:24.897248 7110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.61:46211 every 8 connection(s)
I20260619 17:01:24.898785 7044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/data/info.pb
I20260619 17:01:24.900012 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 7044
I20260619 17:01:24.900240 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-1/wal/instance
I20260619 17:01:24.900799 16576 external_mini_cluster.cc:1428] Setting key d15349c48abcb8210257287e5fefd608
I20260619 17:01:24.904078 7111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:24.914359 7111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.48.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.48.60@KRBTEST.COM" created.
Jun 19 17:01:24 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:01:24.941874 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.917448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.16.48.62:44601)
0619 17:01:24.917971 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:24.917975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:24.919193 (+ 1218us) server_negotiation.cc:407] Connection header received
0619 17:01:24.921669 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:24.921675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:24.921782 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:24.921920 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:24.923762 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.924574 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.926399 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.926819 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.926981 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:24.933613 (+ 6632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:24.933624 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:24.933632 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:24.933706 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:24.937864 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:24.938966 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:24.938975 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:24.938982 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:24.939166 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:24.939659 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:24.939668 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:24.939674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:24.940102 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:24.940296 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:24.941054 (+ 758us) server_negotiation.cc:299] Negotiation successful
0619 17:01:24.941334 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":302,"threads_started":1}
I20260619 17:01:24.942762 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.918133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44601 (local address 127.0.0.1:33302)
0619 17:01:24.919000 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.919048 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:24.921322 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:24.922214 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:24.922224 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:24.922822 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:24.923494 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:24.923514 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.924761 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.924772 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:24.926200 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:24.926214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.927097 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.927104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:24.927365 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:24.928161 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:24.928198 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:24.933245 (+ 5047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:24.938227 (+ 4982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:24.938237 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:24.938253 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:24.938801 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:24.939291 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:24.939295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:24.939297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:24.939530 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:24.940255 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:24.940261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:24.940643 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 17:01:24.940905 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 17:01:24.941327 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":369,"threads_started":1}
I20260619 17:01:24.944789 7111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 17:01:24 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 17:01:24.963492 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.945499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36200)
0619 17:01:24.945707 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.945734 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:24.945838 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:24.947387 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:24.947397 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:24.948001 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:24.948237 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:24.948248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.950385 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.950390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:24.951295 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:24.951343 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.952902 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.952909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:24.953077 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:24.953763 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:24.953782 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:24.955953 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:24.960808 (+ 4855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:24.960813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:24.960815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:24.961233 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:24.961721 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:24.961724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:24.961726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:24.961854 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:24.962610 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:24.962622 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:24.962762 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 17:01:24.962896 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 17:01:24.963127 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 17:01:24.963872 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.945882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.16.48.61:46211)
0619 17:01:24.946550 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:24.946556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:24.946583 (+ 27us) server_negotiation.cc:407] Connection header received
0619 17:01:24.946631 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:24.946636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:24.946831 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:24.947034 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:24.948443 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.950215 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.951531 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:24.952701 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:24.952858 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:24.956263 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:24.956284 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:24.956310 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:24.956364 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:24.960614 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:24.961403 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:24.961416 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:24.961423 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:24.961607 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:24.961956 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:24.961959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:24.961961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:24.962427 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:24.962640 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:24.963095 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 17:01:24.963335 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":366,"threads_started":1}
I20260619 17:01:24.966689 7111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } has no permanent_uuid. Determining permanent_uuid...
I20260619 17:01:24.968014 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.967469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60802)
0619 17:01:24.967643 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.967768 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 17:01:24.968400 7063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
I20260619 17:01:24.970295 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:24.969760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60816)
0619 17:01:24.969934 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:24.970039 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 17:01:24.970542 7111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:35681: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.48.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab.
Entry for principal kudu/127.16.48.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab.
I20260619 17:01:24.971964 16576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
/tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.60:35681
--webserver_interface=127.16.48.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab
--principal=kudu/127.16.48.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.48.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.48.62:44601,127.16.48.61:46211,127.16.48.60:35681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 17:01:25.019935 7111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } attempt: 1
I20260619 17:01:25.021291 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.020686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60830)
0619 17:01:25.020941 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.021086 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 17:01:25.022544 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.022097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60840)
0619 17:01:25.022272 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.022391 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 17:01:25.022890 7111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:35681: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
I20260619 17:01:25.059803 7037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } attempt: 5
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 17:01:25.075716 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.060650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36202)
0619 17:01:25.060875 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.060901 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:25.061016 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:25.061374 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:25.061377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:25.061793 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:25.061972 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.061981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.063392 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.063396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.064485 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.064499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.065533 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.065538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.065684 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.066397 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:25.066416 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:25.068642 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:25.072933 (+ 4291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.072939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.072943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.073408 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.073921 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.073925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.073927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.074098 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.074780 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:25.074784 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:25.074942 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 17:01:25.075079 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 17:01:25.075460 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 17:01:25.075771 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.060766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.16.48.61:46211)
0619 17:01:25.060985 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.060991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.061043 (+ 52us) server_negotiation.cc:407] Connection header received
0619 17:01:25.061117 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.061122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.061221 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.061336 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.062155 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.063242 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.064656 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.065365 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.065506 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.068917 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.068924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.068929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.068976 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.072697 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.073574 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.073580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.073584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.073749 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.074232 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.074237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.074240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.074630 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.074785 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.075209 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.075463 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 17:01:25.077064 7037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } has no permanent_uuid. Determining permanent_uuid...
I20260619 17:01:25.078423 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.077972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60856)
0619 17:01:25.078173 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.078259 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 17:01:25.079636 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.079265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60868)
0619 17:01:25.079410 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.079503 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 17:01:25.079941 7037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:35681: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
I20260619 17:01:25.091167 7111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } attempt: 2
I20260619 17:01:25.092499 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.091873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60884)
0619 17:01:25.092166 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.092287 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 17:01:25.093757 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.093314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60886)
0619 17:01:25.093490 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.093589 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 17:01:25.094043 7111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:35681: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
I20260619 17:01:25.131220 7037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } attempt: 1
I20260619 17:01:25.132633 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.132059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60888)
0619 17:01:25.132328 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.132432 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260619 17:01:25.133884 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.133502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60898)
0619 17:01:25.133669 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.133758 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 17:01:25.134172 7037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:35681: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
I20260619 17:01:25.185305 7111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } attempt: 3
I20260619 17:01:25.187204 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.186533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60914)
0619 17:01:25.186835 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.186959 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 17:01:25.188907 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.188348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60926)
0619 17:01:25.188630 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.188737 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260619 17:01:25.189400 7111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:35681: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
I20260619 17:01:25.202394 7037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } attempt: 2
I20260619 17:01:25.203810 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.203221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60930)
0619 17:01:25.203445 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.203568 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 17:01:25.205255 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.204713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60944)
0619 17:01:25.204920 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.205050 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 17:01:25.205511 7037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:35681: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
I20260619 17:01:25.311756 7037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } attempt: 3
I20260619 17:01:25.313047 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.312461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60950)
0619 17:01:25.312709 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.312823 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 17:01:25.314332 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.313781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60960)
0619 17:01:25.314050 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.314165 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260619 17:01:25.314566 7037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:35681: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
I20260619 17:01:25.332705 7111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } attempt: 4
I20260619 17:01:25.334076 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.333516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60968)
0619 17:01:25.333806 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.333900 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 17:01:25.335466 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.334968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60970)
0619 17:01:25.335257 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.335345 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260619 17:01:25.335727 7111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.48.60:35681: Network error: Client connection negotiation failed: client connection to 127.16.48.60:35681: connect: Connection refused (error 111)
W20260619 17:01:25.359642 7119 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 17:01:25.360009 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 17:01:25.360172 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 17:01:25.360325 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 17:01:25.368254 7119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 17:01:25.368374 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 17:01:25.368462 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 17:01:25.368541 7119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 17:01:25.368633 7119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 17:01:25.378996 7119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.48.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.48.62:44601,127.16.48.61:46211,127.16.48.60:35681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/krb5kdc/kudu_127.16.48.60.keytab
--principal=kudu/127.16.48.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.48.60:35681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.48.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 17:01:25.380568 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 17:01:25.382050 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 17:01:25.395000 7127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:25.395015 7124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 17:01:25.395015 7125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 17:01:25.396227 7119 server_base.cc:1034] running on GCE node
I20260619 17:01:25.397426 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 17:01:25.399049 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 17:01:25.400403 7119 hybrid_clock.cc:648] HybridClock initialized: now 1781888485400311 us; error 77 us; skew 500 ppm
I20260619 17:01:25.400856 7119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888485, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 17:01:25.404662 7119 init.cc:377] Logged in from keytab as kudu/127.16.48.60@KRBTEST.COM (short username kudu)
I20260619 17:01:25.406535 7119 webserver.cc:460] Webserver started at http://127.16.48.60:38833/ using document root <none> and password file <none>
I20260619 17:01:25.407407 7119 fs_manager.cc:359] Metadata directory not provided
I20260619 17:01:25.407516 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 17:01:25.407814 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 17:01:25.411176 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/data/instance:
uuid: "c9f0cfa37eec41b888dba1e3273918ce"
format_stamp: "Formatted at 2026-06-19 17:01:25 on dist-test-slave-3ffd"
server_key: "ff8b98752068e55c7c84d7321713c00c"
server_key_iv: "f651e1f963e9f16632ccae6cf76f46f1"
server_key_version: "encryptionkey@0"
I20260619 17:01:25.412082 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/wal/instance:
uuid: "c9f0cfa37eec41b888dba1e3273918ce"
format_stamp: "Formatted at 2026-06-19 17:01:25 on dist-test-slave-3ffd"
server_key: "ff8b98752068e55c7c84d7321713c00c"
server_key_iv: "f651e1f963e9f16632ccae6cf76f46f1"
server_key_version: "encryptionkey@0"
I20260619 17:01:25.418095 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 17:01:25.422303 7134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 17:01:25.424022 7119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 17:01:25.424310 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/data,/tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/wal
uuid: "c9f0cfa37eec41b888dba1e3273918ce"
format_stamp: "Formatted at 2026-06-19 17:01:25 on dist-test-slave-3ffd"
server_key: "ff8b98752068e55c7c84d7321713c00c"
server_key_iv: "f651e1f963e9f16632ccae6cf76f46f1"
server_key_version: "encryptionkey@0"
I20260619 17:01:25.424552 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 17:01:25.438812 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 17:01:25.443665 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 17:01:25.443914 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 17:01:25.463835 7119 rpc_server.cc:307] RPC server started. Bound to: 127.16.48.60:35681
I20260619 17:01:25.463855 7185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.48.60:35681 every 8 connection(s)
I20260619 17:01:25.465418 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/data/info.pb
I20260619 17:01:25.469506 7186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 17:01:25.472615 16576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu as pid 7119
I20260619 17:01:25.472831 16576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbQ4sJq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781888107067647-16576-0/minicluster-data/master-2/wal/instance
I20260619 17:01:25.473325 16576 external_mini_cluster.cc:1428] Setting key d5a1b25f0a42cf7656aefd183d39ea26
I20260619 17:01:25.478111 7186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888485, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:01:25.491832 7037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } attempt: 4
I20260619 17:01:25.500761 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.481156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.16.48.62:44601)
0619 17:01:25.481769 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.481775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.482369 (+ 594us) server_negotiation.cc:407] Connection header received
0619 17:01:25.482575 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.482579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.482659 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.482762 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.484257 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.485225 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.486947 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.487650 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.487819 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.491989 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.492025 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.492031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.492168 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.495375 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.496465 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.496471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.496474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.496672 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.497200 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.497206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.497209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.497605 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.497768 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.500010 (+ 2242us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.500305 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":287,"threads_started":1}
I20260619 17:01:25.501045 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.475467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.16.48.60:35681)
0619 17:01:25.476381 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.476395 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.476442 (+ 47us) server_negotiation.cc:407] Connection header received
0619 17:01:25.478278 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.478304 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.478803 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.479251 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.480893 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.483112 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.484847 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.485866 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.486071 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.491068 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.491080 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.491099 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.491138 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.496384 (+ 5246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.497410 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.497420 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.497423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.497588 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.498087 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.498092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.498095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.498579 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.498915 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.499302 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.499538 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"spinlock_wait_cycles":24832,"thread_start_us":348,"threads_started":1}
I20260619 17:01:25.500953 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.481186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.62:44601 (local address 127.0.0.1:33308)
0619 17:01:25.482176 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.482240 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:25.482422 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:25.482872 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:25.482879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:25.483700 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:25.483962 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.483972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.485412 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.485418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.486779 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.486791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.487944 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.487950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.488085 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.488934 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:25.488965 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:25.491692 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:25.495556 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.495571 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.495589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.496320 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.496829 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.496834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.496838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.497057 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.497837 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:25.497844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:25.498335 (+ 491us) client_negotiation.cc:769] Sending connection context
0619 17:01:25.498711 (+ 376us) client_negotiation.cc:240] Negotiation successful
0619 17:01:25.499291 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":532,"threads_started":1}
I20260619 17:01:25.503152 7186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.62@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888485, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 17:01:25.513306 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.492849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.16.48.60:35681)
0619 17:01:25.496336 (+ 3487us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.496342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.496363 (+ 21us) server_negotiation.cc:407] Connection header received
0619 17:01:25.496438 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.496443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.496524 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.496652 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.497611 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.498772 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.500686 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.501477 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.502320 (+ 843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.505986 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.505996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.506001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.506107 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.510032 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.510985 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.510991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.510997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.511177 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.511695 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.511700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.511703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.512069 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.512313 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.512787 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.513003 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3359,"thread_start_us":3088,"threads_started":1}
I20260619 17:01:25.513442 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.492563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60982)
0619 17:01:25.493210 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.493234 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:25.493622 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:25.496717 (+ 3095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:25.496724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:25.497193 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:25.497442 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.497452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.498946 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.498951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.500470 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.500487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.501671 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.501677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.501921 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.502695 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:25.502723 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:25.505764 (+ 3041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:25.510197 (+ 4433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.510206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.510212 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.510842 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.511318 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.511326 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.511332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.511552 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.512298 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:25.512303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:25.512504 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 17:01:25.512680 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 17:01:25.513033 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.62@KRBTEST.COM
I20260619 17:01:25.520310 7037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f: Bootstrap starting.
I20260619 17:01:25.520560 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.504341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.16.48.61:46211)
0619 17:01:25.504561 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.504576 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.504598 (+ 22us) server_negotiation.cc:407] Connection header received
0619 17:01:25.504667 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.504675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.504762 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.504862 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.506048 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.506879 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.508870 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.509318 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.509478 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.513467 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.513475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.513481 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.513537 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.517131 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.517931 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.517938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.517941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.518097 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.518617 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.518623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.518627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.519017 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.519187 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.519765 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.520032 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 17:01:25.521157 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.503845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.61:46211 (local address 127.0.0.1:36204)
0619 17:01:25.504059 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.504082 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:25.504242 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:25.505075 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:25.505100 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:25.505601 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:25.505810 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.505825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.507039 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.507045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.508696 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.508717 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.509486 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.509491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.509684 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.510277 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:25.510300 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:25.513164 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:25.517322 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.517329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.517333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.517773 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.518245 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.518250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.518254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.518462 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.519254 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:25.519259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:25.519451 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 17:01:25.520193 (+ 742us) client_negotiation.cc:240] Negotiation successful
0619 17:01:25.520614 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 17:01:25.522300 7186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } has no permanent_uuid. Determining permanent_uuid...
I20260619 17:01:25.524724 7037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:25.525429 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.504927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.16.48.62:44601)
0619 17:01:25.505131 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.505137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.505227 (+ 90us) server_negotiation.cc:407] Connection header received
0619 17:01:25.505419 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.505424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.505525 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.505634 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.507274 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.508493 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.510651 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.511273 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.511391 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.518502 (+ 7111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.518513 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.518519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.518571 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.522042 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.523098 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.523105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.523108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.523293 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.523738 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.523748 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.523752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.524086 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.524456 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.524793 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.525044 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 17:01:25.526084 7037 log.cc:824] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:25.530045 7037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f: No bootstrap required, opened a new log
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888485, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.60@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.48.61@KRBTEST.COM
I20260619 17:01:25.540074 7037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } }
I20260619 17:01:25.540634 7037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3638c182c9bb416da6efbff28026102f, State: Initialized, Role: FOLLOWER
I20260619 17:01:25.541215 7037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } }
I20260619 17:01:25.542397 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.542902 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:25.543773 7037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:25.543854 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.523335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.16.48.60:35681)
0619 17:01:25.526728 (+ 3393us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.526736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.526756 (+ 20us) server_negotiation.cc:407] Connection header received
0619 17:01:25.526819 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.526824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.526919 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.527061 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.528043 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.529305 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.530579 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.531174 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.531905 (+ 731us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.535480 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.535489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.535494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.535540 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.539445 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.541177 (+ 1732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.541185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.541190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.541367 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.541914 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.541921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.541926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.542371 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.542527 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.543118 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.543427 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3227}
I20260619 17:01:25.544937 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.523070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:60994)
0619 17:01:25.523289 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.523310 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:25.523441 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:25.527131 (+ 3690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:25.527137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:25.527656 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:25.527869 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.527883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.529540 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.529546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.530423 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.530435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.531336 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.531342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.531553 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.532371 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:25.532401 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:25.535021 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:25.540311 (+ 5290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.540316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.540319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.541000 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.541544 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.541552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.541558 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.541760 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.542629 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:25.542634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:25.542815 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 17:01:25.544082 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0619 17:01:25.544477 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 17:01:25.549939 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.527184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.16.48.61:46211)
0619 17:01:25.527374 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.527379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.527399 (+ 20us) server_negotiation.cc:407] Connection header received
0619 17:01:25.527575 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.527580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.527662 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.527780 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.529586 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.530665 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.532553 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.533262 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.533409 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.539725 (+ 6316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.539733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.539737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.539786 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.543769 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.545041 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.545050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.545056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.545276 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.545803 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.545809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.545817 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.546239 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.548534 (+ 2295us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.548995 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.549406 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 17:01:25.551214 7186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce: Bootstrap starting.
I20260619 17:01:25.556224 7186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:25.557137 16576 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 17:01:25.557725 7186 log.cc:824] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:25.561271 7186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce: No bootstrap required, opened a new log
I20260619 17:01:25.561715 7012 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3638c182c9bb416da6efbff28026102f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33316
I20260619 17:01:25.562131 7012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 17:01:25.562338 7012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:25.565371 7012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } }
I20260619 17:01:25.568320 7012 leader_election.cc:288] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [CANDIDATE]: Term 1 election: Requested vote from peers 03aad3aeace2409687ee768ac0365e4c (127.16.48.61:46211), c9f0cfa37eec41b888dba1e3273918ce (127.16.48.60:35681)
I20260619 17:01:25.572633 7186 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } }
I20260619 17:01:25.573302 7186 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0cfa37eec41b888dba1e3273918ce, State: Initialized, Role: FOLLOWER
I20260619 17:01:25.574101 7186 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } }
I20260619 17:01:25.576923 7209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.577517 7209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [sys.catalog]: This master's current role is: FOLLOWER
W20260619 17:01:25.578111 6991 tablet.cc:2369] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:01:25.581676 7186 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:25.581806 7161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3638c182c9bb416da6efbff28026102f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c9f0cfa37eec41b888dba1e3273918ce"
I20260619 17:01:25.582623 7161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:25.587611 7161 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3638c182c9bb416da6efbff28026102f in term 1.
I20260619 17:01:25.588768 6989 leader_election.cc:302] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3638c182c9bb416da6efbff28026102f, c9f0cfa37eec41b888dba1e3273918ce; no voters:
I20260619 17:01:25.589501 7194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [term 1 FOLLOWER]: Leader election won for term 1
W20260619 17:01:25.592628 7206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 17:01:25.592794 7206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 17:01:25.574276 6970 master_runner.cc:186] Error getting master registration for 127.16.48.61:46211: OK, instance_id { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" instance_seqno: 1781888484891342 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 17:01:25.610613 6970 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 17:01:25.610813 7194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [term 1 LEADER]: Becoming Leader. State: Replica: 3638c182c9bb416da6efbff28026102f, State: Running, Role: LEADER
I20260619 17:01:25.611459 7194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } }
I20260619 17:01:25.631117 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3638c182c9bb416da6efbff28026102f. Latest consensus state: current_term: 1 leader_uuid: "3638c182c9bb416da6efbff28026102f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.631789 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:25.633157 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.593479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.16.48.60:35681)
0619 17:01:25.593863 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.593873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.596844 (+ 2971us) server_negotiation.cc:407] Connection header received
0619 17:01:25.597517 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.597526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.597652 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.597747 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.601394 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.602932 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.611203 (+ 8271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.611972 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.612176 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.617444 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.617452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.617458 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.617529 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.621483 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.626880 (+ 5397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.626892 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.626898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.627186 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.627834 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.627843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.627849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.628411 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.628831 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.632190 (+ 3359us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.632581 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 17:01:25.634949 7111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } attempt: 5
I20260619 17:01:25.636854 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.594966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.16.48.61:46211)
0619 17:01:25.595156 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.595161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.601133 (+ 5972us) server_negotiation.cc:407] Connection header received
0619 17:01:25.601339 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.601348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.601470 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.601605 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.605959 (+ 4354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.607250 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.611571 (+ 4321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.612598 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.612757 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.619798 (+ 7041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.619809 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.619817 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.619890 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.624748 (+ 4858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.629548 (+ 4800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.629570 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.629577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.629786 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.630423 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.630432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.630438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.630985 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.631338 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.636171 (+ 4833us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.636464 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 17:01:25.646474 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.593540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.16.48.62:44601)
0619 17:01:25.593721 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.593728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.595137 (+ 1409us) server_negotiation.cc:407] Connection header received
0619 17:01:25.595663 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.595672 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.595760 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.595835 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.600459 (+ 4624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.607368 (+ 6909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.626123 (+ 18755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.626800 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.626945 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.632214 (+ 5269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.632222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.632227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.632275 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.636857 (+ 4582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.638668 (+ 1811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.638675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.638679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.638988 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.639630 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.639665 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.639669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.640469 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.642109 (+ 1640us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.642441 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.643603 (+ 1162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 17:01:25 dist-test-slave-3ffd krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.48.61@KRBTEST.COM for kudu/127.16.48.60@KRBTEST.COM
I20260619 17:01:25.652781 7223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 17:01:25.655498 7223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 17:01:25.666234 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.635582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.48.60:35681 (local address 127.0.0.1:32782)
0619 17:01:25.635972 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 17:01:25.635995 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 17:01:25.636156 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 17:01:25.636856 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 17:01:25.636862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 17:01:25.637361 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 17:01:25.637585 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.637599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.639346 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.639355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.641358 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 17:01:25.641383 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.642920 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.642930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 17:01:25.643291 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.644219 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 17:01:25.644245 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 17:01:25.647785 (+ 3540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 17:01:25.659676 (+ 11891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.659684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.659689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.660420 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.660963 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 17:01:25.660969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 17:01:25.660973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 17:01:25.661166 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 17:01:25.661951 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 17:01:25.661957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 17:01:25.662187 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 17:01:25.662337 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 17:01:25.665779 (+ 3442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260619 17:01:25.667968 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:25.636360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.16.48.60:35681)
0619 17:01:25.636524 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.636530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.636546 (+ 16us) server_negotiation.cc:407] Connection header received
0619 17:01:25.636603 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.636610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.636682 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.637767 (+ 1085us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.637942 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.639138 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.641643 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:25.642682 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:25.643763 (+ 1081us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:25.648322 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 17:01:25.648337 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 17:01:25.648375 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 17:01:25.648552 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0619 17:01:25.659240 (+ 10688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.660619 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.660625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.660629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.660803 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 17:01:25.661328 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 17:01:25.661334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 17:01:25.661338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 17:01:25.661785 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 17:01:25.667045 (+ 5260us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:25.667235 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 17:01:25.667592 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 17:01:25.675969 7161 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [term 1 FOLLOWER]: Refusing update from remote peer 3638c182c9bb416da6efbff28026102f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:01:25.677244 7215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:01:25.677501 7111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c: Bootstrap starting.
W20260619 17:01:25.696497 7140 tablet.cc:2369] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 17:01:25.717562 7111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c: Neither blocks nor log segments found. Creating new log.
I20260619 17:01:25.718067 7226 mvcc.cc:205] Tried to move back new op lower bound from 7298615237303115776 to 7298615237072105472. Current Snapshot: MvccSnapshot[applied={T|T < 7298615237303115776}]
I20260619 17:01:25.719292 7111 log.cc:824] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c: Log is configured to *not* fsync() on all Append() calls
I20260619 17:01:25.721359 7209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3638c182c9bb416da6efbff28026102f. Latest consensus state: current_term: 1 leader_uuid: "3638c182c9bb416da6efbff28026102f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.721823 7209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:25.733770 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3638c182c9bb416da6efbff28026102f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.735822 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:25.741837 7223 catalog_manager.cc:1347] Generated new cluster ID: dbf3c4a3f84040c5a94717f0f55712ae
I20260619 17:01:25.742003 7223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 17:01:25.742746 7209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3638c182c9bb416da6efbff28026102f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.743079 7209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce [sys.catalog]: This master's current role is: FOLLOWER
W20260619 17:01:25.758649 7230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 17:01:25.758857 7230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 17:01:25.706813 7119 master_runner.cc:186] Error getting master registration for 127.16.48.61:46211: OK, instance_id { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" instance_seqno: 1781888484891342 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 17:01:25.777328 7119 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 17:01:25.784771 7111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c: No bootstrap required, opened a new log
I20260619 17:01:25.791637 7223 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 17:01:25.795315 7223 catalog_manager.cc:1504] Loading token signing keys...
I20260619 17:01:25.802695 7111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } }
I20260619 17:01:25.803224 7111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03aad3aeace2409687ee768ac0365e4c, State: Initialized, Role: FOLLOWER
I20260619 17:01:25.803810 7111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } }
I20260619 17:01:25.808797 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.809311 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:25.811497 7111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 17:01:25.811560 7086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3638c182c9bb416da6efbff28026102f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03aad3aeace2409687ee768ac0365e4c"
I20260619 17:01:25.811964 7085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [term 0 FOLLOWER]: Advancing to term 1
I20260619 17:01:25.817325 7085 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [term 1 FOLLOWER]: Refusing update from remote peer 3638c182c9bb416da6efbff28026102f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 17:01:25.822389 7086 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3638c182c9bb416da6efbff28026102f in term 1.
I20260619 17:01:25.825886 7194 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [LEADER]: Connected to new peer: Peer: permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 17:01:25.836664 7223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f: Generated new TSK 0
I20260619 17:01:25.842705 7223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 17:01:25.892272 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3638c182c9bb416da6efbff28026102f. Latest consensus state: current_term: 1 leader_uuid: "3638c182c9bb416da6efbff28026102f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.892665 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:25.893405 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3638c182c9bb416da6efbff28026102f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.893862 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 17:01:25.901619 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3638c182c9bb416da6efbff28026102f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638c182c9bb416da6efbff28026102f" member_type: VOTER last_known_addr { host: "127.16.48.62" port: 44601 } } peers { permanent_uuid: "03aad3aeace2409687ee768ac0365e4c" member_type: VOTER last_known_addr { host: "127.16.48.61" port: 46211 } } peers { permanent_uuid: "c9f0cfa37eec41b888dba1e3273918ce" member_type: VOTER last_known_addr { host: "127.16.48.60" port: 35681 } } }
I20260619 17:01:25.902019 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3638c182c9bb416da6efbff28026102f [sys.catalog]: This master's current role is: LEADER
I20260619 17:01:25.903796 7254 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 17:01:25.970990 7190 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:25.958643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.16.48.62:44601)
0619 17:01:25.958898 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.958907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.968865 (+ 9958us) server_negotiation.cc:407] Connection header received
0619 17:01:25.969179 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.969185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.969286 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.969362 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.970710 (+ 1348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33332: BlockingRecv error: recv got EOF from 127.0.0.1:33332 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 17:01:25.973678 7192 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:25.958297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.16.48.60:35681)
0619 17:01:25.958494 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.958503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.970695 (+ 12192us) server_negotiation.cc:407] Connection header received
0619 17:01:25.971974 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.971981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.972070 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.972187 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.973383 (+ 1196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32794: BlockingRecv error: recv got EOF from 127.0.0.1:32794 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 17:01:25.973881 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:25.971345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.16.48.61:46211)
0619 17:01:25.971651 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.971660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.971913 (+ 253us) server_negotiation.cc:407] Connection header received
0619 17:01:25.972190 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.972197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.972284 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.972356 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.973646 (+ 1290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36228: BlockingRecv error: recv got EOF from 127.0.0.1:36228 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
W20260619 17:01:25.980846 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:25.978404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.16.48.61:46211)
0619 17:01:25.978600 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.978607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.979144 (+ 537us) server_negotiation.cc:407] Connection header received
0619 17:01:25.979383 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.979388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.979529 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.979652 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.980585 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36244: BlockingRecv error: recv got EOF from 127.0.0.1:36244 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 17:01:25.989861 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:25.985678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.16.48.61:46211)
0619 17:01:25.985866 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:25.985874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:25.988138 (+ 2264us) server_negotiation.cc:407] Connection header received
0619 17:01:25.988380 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:25.988387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:25.988491 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:25.988630 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:25.989618 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36254: BlockingRecv error: recv got EOF from 127.0.0.1:36254 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 17:01:26.005005 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:26.002494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.16.48.61:46211)
0619 17:01:26.002694 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:26.002700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:26.003279 (+ 579us) server_negotiation.cc:407] Connection header received
0619 17:01:26.003497 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:26.003503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:26.003626 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:26.003744 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:26.004745 (+ 1001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36260: BlockingRecv error: recv got EOF from 127.0.0.1:36260 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 17:01:26.021000 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:26.018471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.16.48.61:46211)
0619 17:01:26.018726 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:26.018732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:26.019320 (+ 588us) server_negotiation.cc:407] Connection header received
0619 17:01:26.019524 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:26.019532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:26.019664 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:26.019794 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:26.020756 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36272: BlockingRecv error: recv got EOF from 127.0.0.1:36272 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260619 17:01:26.035681 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:26.033151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.16.48.61:46211)
0619 17:01:26.033352 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:26.033358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:26.033917 (+ 559us) server_negotiation.cc:407] Connection header received
0619 17:01:26.034155 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:26.034162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:26.034276 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:26.034401 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:26.035367 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36278: BlockingRecv error: recv got EOF from 127.0.0.1:36278 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 17:01:26.061286 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:26.058434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.16.48.61:46211)
0619 17:01:26.058648 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:26.058655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:26.059210 (+ 555us) server_negotiation.cc:407] Connection header received
0619 17:01:26.059533 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:26.059540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:26.059644 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:26.059726 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:26.061020 (+ 1294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36292: BlockingRecv error: recv got EOF from 127.0.0.1:36292 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 17:01:26.099821 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:26.097391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.16.48.61:46211)
0619 17:01:26.097601 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:26.097607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:26.098161 (+ 554us) server_negotiation.cc:407] Connection header received
0619 17:01:26.098370 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:26.098376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:26.098498 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:26.098637 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:26.099529 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36300: BlockingRecv error: recv got EOF from 127.0.0.1:36300 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 17:01:26.179399 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:26.176923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.16.48.61:46211)
0619 17:01:26.177138 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:26.177145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:26.177732 (+ 587us) server_negotiation.cc:407] Connection header received
0619 17:01:26.177986 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:26.177992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:26.178095 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:26.178221 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:26.179161 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36310: BlockingRecv error: recv got EOF from 127.0.0.1:36310 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 17:01:26.313277 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:26.311108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.16.48.61:46211)
0619 17:01:26.311394 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:26.311401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:26.311641 (+ 240us) server_negotiation.cc:407] Connection header received
0619 17:01:26.311913 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:26.311922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:26.312026 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:26.312144 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:26.312984 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36320: BlockingRecv error: recv got EOF from 127.0.0.1:36320 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260619 17:01:26.575380 7117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 17:01:26.573093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.16.48.61:46211)
0619 17:01:26.573445 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:26.573454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:26.573879 (+ 425us) server_negotiation.cc:407] Connection header received
0619 17:01:26.574129 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:26.574133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:26.574212 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:26.574336 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 17:01:26.575162 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36326: BlockingRecv error: recv got EOF from 127.0.0.1:36326 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
I20260619 17:01:26.760917 7230 catalog_manager.cc:1259] Loaded cluster ID: dbf3c4a3f84040c5a94717f0f55712ae
I20260619 17:01:26.761034 7230 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce: loading cluster ID for follower catalog manager: success
I20260619 17:01:26.763598 7230 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce: acquiring CA information for follower catalog manager: success
I20260619 17:01:26.765570 7230 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9f0cfa37eec41b888dba1e3273918ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 17:01:26.910543 7254 catalog_manager.cc:1259] Loaded cluster ID: dbf3c4a3f84040c5a94717f0f55712ae
I20260619 17:01:26.910698 7254 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c: loading cluster ID for follower catalog manager: success
I20260619 17:01:26.914089 7254 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c: acquiring CA information for follower catalog manager: success
I20260619 17:01:26.916425 7254 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03aad3aeace2409687ee768ac0365e4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 17:01:27.103659 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:27.094082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.16.48.61:46211)
0619 17:01:27.097198 (+ 3116us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:27.097203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:27.097221 (+ 18us) server_negotiation.cc:407] Connection header received
0619 17:01:27.097290 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:27.097294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:27.097458 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:27.097624 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:27.098118 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.099188 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.100978 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.101487 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.101667 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:27.102255 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:27.102891 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:27.103091 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:27.103209 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 17:01:27.103312 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2969,"thread_start_us":2804,"threads_started":1}
I20260619 17:01:27.119428 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:27.111404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.16.48.60:35681)
0619 17:01:27.112068 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:27.112075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:27.112422 (+ 347us) server_negotiation.cc:407] Connection header received
0619 17:01:27.112687 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:27.112693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:27.112930 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:27.113133 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:27.113606 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.114710 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.116577 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.117196 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.117324 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:27.117944 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:27.118698 (+ 754us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:27.118865 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:27.118985 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 17:01:27.119091 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":273,"threads_started":1}
I20260619 17:01:27.150512 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:27.127160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.16.48.60:35681)
0619 17:01:27.133547 (+ 6387us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:27.133554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:27.141326 (+ 7772us) server_negotiation.cc:407] Connection header received
0619 17:01:27.141691 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:27.141698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:27.141815 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:27.141885 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:27.143326 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.144492 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.147884 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.148604 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.148712 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:27.149627 (+ 915us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:27.149782 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:27.149850 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:27.150097 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 17:01:27.150199 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6205}
I20260619 17:01:27.150701 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:27.141448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.16.48.61:46211)
0619 17:01:27.141670 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:27.141676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:27.142945 (+ 1269us) server_negotiation.cc:407] Connection header received
0619 17:01:27.143132 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:27.143138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:27.143238 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:27.143349 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:27.144204 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.145382 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.148169 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.148919 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.149033 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:27.149787 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:27.149943 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:27.150011 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:27.150236 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 17:01:27.150339 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 17:01:27.151901 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 17:01:27.127364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.16.48.62:44601)
0619 17:01:27.139549 (+ 12185us) server_negotiation.cc:206] Beginning negotiation
0619 17:01:27.139558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 17:01:27.141146 (+ 1588us) server_negotiation.cc:407] Connection header received
0619 17:01:27.141294 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 17:01:27.141300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 17:01:27.141557 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 17:01:27.141755 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 17:01:27.143208 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.144504 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.147874 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 17:01:27.148587 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 17:01:27.148724 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 17:01:27.149706 (+ 982us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 17:01:27.150201 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 17:01:27.150301 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 17:01:27.151051 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 17:01:27.151575 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11977,"thread_start_us":11441,"threads_started":1}
I20260619 17:01:27.159772 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 6970
I20260619 17:01:27.161155 7032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:01:27.278548 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 6970
I20260619 17:01:27.300858 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 7044
I20260619 17:01:27.302114 7106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:01:27.414887 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 7044
I20260619 17:01:27.436340 16576 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu pid 7119
I20260619 17:01:27.437626 7181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 17:01:27.546727 16576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbQ4sJq/build/asan/bin/kudu with pid 7119
2026-06-19T17:01:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5108 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10204 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (380430 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST